CA Runscope Review

Integration with alerting apps tightens our feedback loop during testing


What is our primary use case?

API monitoring for IoT data pipeline in the cloud.

How has it helped my organization?

This solution has improved our responsiveness to issues in both production and non-production environments.

Through integration with Slack, our development and DevOps teams are quickly aware of issues related to new changes deployed to non-production environments, which tightens our feedback loop during testing. Through integration with PagerDuty, our DevOps resources are able to reduce the adverse impact of a given production issue.

Runscope has also helped to expose subtle network latency or routing issues with components managed by our cloud provider.

What is most valuable?

API monitoring where specific tests are designed to emulate production data-flows through our pipelines. Tests are per-step configurable with script capabilities and support for token-based authentication using variables. Configurable test frequency and integration with both Slack and PagerDuty for alerting our DevOps team are also important. General use of the dashboard by multiple team members is also valuable.

What needs improvement?

  1. Ability to configure Runscope behavior when a given test has its schedule modified. That is, in a non-production environment, we often need to pause a test (either by removing it from the schedule or adjusting its frequency) and this results in the automatic kick-off of this test in all other environments. We would like the ability to simply pause a test for a specific shared environment without causing additional tests to be started.
  2. We definitely make use of historical test results, especially when tracking more difficult issues such as timeouts. Per our test frequency (15 minutes) we often need to go back further than the last 100 test runs. We are aware that results further back in time are available through the Runscope API, however, it would be great if the dashboard allowed easy access to a larger amount of historical data.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

There have been some Runscope outages in the last year. Though not many, Runscope availability is of critical importance in our production context. This was the primary reason for the "recommend this solution" score of eight rather than nine or 10.

What do I think about the scalability of the solution?

No issues with scalability.

How is customer service and technical support?

Tech support is responsive through website chat (this is the only communication channel we have used).

How was the initial setup?

Setup was fairly straightforward.

What's my experience with pricing, setup cost, and licensing?

Having the different plans organized by team size, buckets, and requests per billing period works for our needs. Larger companies may need to take a close look at these elements.

Which other solutions did I evaluate?

Some alternatives were evaluated prior to choosing Runscope, however, I do not have access to that information at this time.

What other advice do I have?

Create a DevOps team story to evaluate Runscope, which includes setup for all tests/buckets which will be used in your production environment. Exercise all use cases and outcomes, including integration with your desired tools (e.g. Slack, PagerDuty, etc.), and tune the available configurations as needed.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
Add a Comment
Guest
Sign Up with Email