We just raised a $30M Series A: Read our story

Runscope OverviewUNIXBusinessApplication

What is Runscope?

Runscope is also known as CA Runscope.

Runscope Customers

Twilio, Microsoft, Adobe, Okta, Tesco, Charter Communications

Runscope Video

Archived Runscope Reviews (more than two years old)

Filter by:
Filter Reviews
Industry
Loading...
Filter Unavailable
Company Size
Loading...
Filter Unavailable
Job Level
Loading...
Filter Unavailable
Rating
Loading...
Filter Unavailable
Considered
Loading...
Filter Unavailable
Order by:
Loading...
  • Date
  • Highest Rating
  • Lowest Rating
  • Review Length
Search:
Showingreviews based on the current filters. Reset all filters
NR
Director of Engineering at a consultancy with 1-10 employees
Real User
Enables us to monitor API endpoints to ensure everything is running

Pros and Cons

  • "It uses JavaScript which is easy."
  • "You can run it off the cloud and against different data centers. That's one of the plus points for most people for this product, as opposed to some other solutions."
  • "I've noticed, once or twice, when there were some updates pushed by Runscope, at times it didn't refresh the pages. I was running it, there were multiple tests running at the same time, and it didn't update the webpage I was looking at. So I had to refresh it manually. Occasionally, it still happens."
  • "the built-in editor for the scripts can be improved a lot. Instead of making such a basic editor, maybe include a little more so it is easier to look at things and debug."

What is our primary use case?

We run all our production-level test cases on it. We use it to run against APIs to make sure those API endpoints are still there, that nothing has happened to them, that they haven't stopped running. This is a way of checking every five minutes. We use it as a monitoring tool.

How has it helped my organization?

It helps make sure that the program running the production server hasn't crashed without anybody knowing. Runscope makes sure that it runs properly and, if it does crash, something happens. It sends an email. Then, with Slack, we can notify somebody that something is not right.

What is most valuable?

It uses JavaScript which is easy.

What needs improvement?

I've noticed, once or twice, when there were some updates pushed by Runscope, at times it didn't refresh the pages. I was running it, there were multiple tests running at the same time, and it didn't update the webpage I was looking at. So I had to refresh it manually. Occasionally, it still happens.

Another thing I have found is that the built-in editor for the scripts can be improved a lot. Instead of making such a basic editor, maybe include a little more so it is easier to look at things and debug.

Finally, other than the Runscope documentation, I haven't seen too much elsewhere. I haven't seen a user community.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

In general, it's pretty stable. Once or twice we have had some issues but that's it.

What do I think about the scalability of the solution?

It is scalable, within the scope of what we're using it for.

How are customer service and technical support?

I have not had an opportunity to contact technical support yet.

Which solution did I use previously and why did I switch?

Although I wasn't with the company when they chose Runscope, one of the main things I was told was that they chose this product is because it's cloud-based. The company has different data centers. You can run it off the cloud and against different data centers. That's one of the plus points for most people for this product, as opposed to some other solutions.

How was the initial setup?

I had a local setup done. It was pretty easy and straightforward.

I set it up in the development environment so that certain tests run there. I also did quite a bit of the production environment setup. That was more like creating an environment, not installing anything, such as a Runscope agent.

It was pretty straightforward. All the documentation is there. You just can't search the documentation.

What other advice do I have?

It's a pretty good product, as long as the price is okay for you.

Disclosure: I am a real user, and this review is based on my own experience and opinions.
CA
Lead QA Engineer at a financial services firm with 51-200 employees
Real User
Enables us to string together a number of API tests for a test suite

Pros and Cons

  • "We love the fact that we can have our API tests run on a schedule as often as we need. We also take advantage of being able to set up different environment settings so that we can use the same test in our production, integration, and QA environments, easily. The ability to string together a number of API tests for a test suite is very important to us as well."
  • "Navigation can be a little tricky when changing run environments and switching between test run results and editing the tests. It would be nice if you could switch the environment for all tests in a Bucket to run, rather than having to change each test environment."
  • "I would like to see easier integration and display of test results so that the data could be shared via a dashboard within our company. (Some integration is possible, provided you use one of the supported platforms)."

What is our primary use case?

We use Runscope to test our API endpoints for our product, both on-demand and via a schedule.

How has it helped my organization?

Not only does Runscope allow us to keep our tests organized and available to everyone on our team, but it also alerts us when tests fail, via integration with Slack, so that we can address bugs that have been introduced.

What is most valuable?

  • We love the fact that we can have our API tests run on a schedule as often as we need.
  • We also take advantage of being able to set up different environment settings so that we can use the same test in our production, integration, and QA environments, easily.
  • The ability to string together a number of API tests for a test suite is very important to us as well.

What needs improvement?

Navigation can be a little tricky when changing run environments and switching between test run results and editing the tests. It would be nice if you could switch the environment for all tests in a Bucket to run, rather than having to change each test environment.

I would also like to see easier integration and display of test results so that the data can be shared via a dashboard within our company. (Some integration is possible, provided you use one of the supported platforms).

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

We have rarely seen any issues and have never seen an outage with Runscope's servers. Just make sure the agent is run on the proper hardware.

What do I think about the scalability of the solution?

We are not limited in the number of tests that we run, nor the frequency, provided that we stay within our contracted rate. It is easy to manage that information.

How are customer service and technical support?

We have had good success with tech support, especially when we were getting started. They are knowledgable and will help you all the way through your process until you understand the solution to your problem.

Which solution did I use previously and why did I switch?

We have used Postman quite a bit, but Runscope makes it easier to group tests together in a logical way. In addition, being able to run tests on a schedule and make them available to everyone on a team is great.

How was the initial setup?

Easy setup, easy to make your first tests. There is a bit of a learning curve when you are trying to decide how to group tests and use variables, but nothing that their tech team isn't able to help you through.

What's my experience with pricing, setup cost, and licensing?

Pricing is very reasonable and is easy to pay when you are supporting a team of engineers.

Which other solutions did I evaluate?

We did not evaluate other options, other than continuing to use Postman for our API testing.

What other advice do I have?

Tests can be brittle, so make sure you include checks to validate data and variables that you are using.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
MY
QA Analyst at a tech services company with 51-200 employees
Real User
Pre-request and Post-response Scripts save a lot of time debugging and revising

What is our primary use case?

We use it for doing smoke tests in production and testing in QA and in team environments.

How has it helped my organization?

It speeds up production deployment.

What is most valuable?

Pre-request Scripts and Post-response Scripts are very handy. Instead of adding a bulk of assertions to those associated test cases one by one, I can easily just write scripts, then copy and paste them into my test cases. This really saves me a lot of time on debugging and revising.

What needs improvement?

The documentation of Runscope needs to be improved. For instance, there is very much a lack of examples in the documentation. It is a bit difficult for beginners and intermediate users to get started quickly or to go deeper when creating test cases.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

All my test cases failed when Runscope did a security patch a few months ago. That was the first time I found the stability wasn't great. Otherwise, it has been doing fine.

What do I think about the scalability of the solution?

I have experienced no issues with scalability.

How are customer service and technical support?

I have not had to contact tech support yet.

Which solution did I use previously and why did I switch?

We had a previous solution, but Runscope has a better user interface and is easy to set up, so that is why we switched.

How was the initial setup?

The initial setup is pretty straightforward.

What's my experience with pricing, setup cost, and licensing?

I am not aware of the pricing and licensing as these topics are not part of my job.

What other advice do I have?

Ask for more examples in the documentation.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
BR
Director, QA
Real User
A cloud-based environment for building and executing tests, allowing me to focus on building, running, and reviewing test results

Pros and Cons

  • "It is a cloud-based environment for building and executing tests, which allows me to focus on building, running, and reviewing test results."
  • "File upload is a big part of the products that we test. The lack of file upload in Runscope requires us to still use UI (Selenium) automated tests for these scenarios."

What is our primary use case?

We use it for API regression tests and API production monitoring.

How has it helped my organization?

It has allowed us to go from no API tests to scheduled, automated tests integrated with PagerDuty in a few months. I was able to do this without having to hire a QA Developer. Existing manual QA testers were able to learn simple JavaScript and Runscope UI. Regression tests were executed in minutes instead of days.

What is most valuable?

It is a cloud-based environment for building and executing tests, which allows me to focus on building, running, and reviewing test results. I do not have to waste time managing development environments on local machines. Being cloud-based, I can check the status on the test results from anywhere on any device. I am not constrained by a physical machine where the tests are running.

What needs improvement?

File upload is a big part of the products that we test. The lack of file upload in Runscope requires us to still use UI (Selenium) automated tests for these scenarios. Having Runscope support file uploads is the biggest improvement that we would benefit from. I would be willing to pay more if file upload was supported.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

Stability has been good. It is not an issue for us.

What do I think about the scalability of the solution?

We have experienced issues with scalability so far. A few months back, I needed to do some load testing which was not what Runscope was designed to do. I was able to hack together a solution in Runscope which provided the data that we needed. This is a complement to its flexibility.

How are customer service and technical support?

I have only contacted tech support a couple of times. The experiences were positive.

Which solution did I use previously and why did I switch?

Before Runscope, I only used UI (Selenium) testing solutions.

How was the initial setup?

The complexity of the setup was due more to the complexity of the API calls in the system that I was testing, not the complexity of Runscope. To make it easier to onboard new customers, CA may want to have example tests with a couple API calls demonstrating assertions and variables.

What's my experience with pricing, setup cost, and licensing?

The pricing plans that we have used so far have been selected purely on volume needed per month.

Which other solutions did I evaluate?

We evaluated Assertible and Postman.

What other advice do I have?

  • Prioritize tests.
  • Use environment variables.
  • Budget time to experiment with features before going full speed on test creation.
Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
LL
Quality Analyst at a tech vendor with 10,001+ employees
Real User
Enabled us to set up automatic regression tests that cover everything we need to monitor

Pros and Cons

  • "Valuable features include: Being able to add tests as sub-tests in a parent test for regression testing; notifications for tests that pass/fail; being able to customize expected results with assertions; being able to set tests to run daily."
  • "We would like to be able to select the actual time a test will run automatically. It appears that you can set up the automatic schedule, but the test runs based on the last time it ran and not a time you select. Since we want these tests to run in the middle of the night, it would be nice to be able to select that."
  • "It would be nice to be able to generate API documentation."

What is our primary use case?

Initial method testing and regression testing.

How has it helped my organization?

We have been able to set up regression tests that cover everything we need to monitor. We then set them to run automatically and forget about them. We are notified whether they pass/fail, and have been able to catch issues quickly when changes were made to our methods.

What is most valuable?

  • Being able to add tests as sub-tests in a parent test for regression testing
  • Notifications for tests that pass/fail
  • Being able to customize expected results with assertions
  • Being able to set tests to run daily
  • The intuitive user interface that makes it easy to set up everything without a programming background

What needs improvement?

We would like to be able to select the actual time a test will run automatically. It appears that you can set up the automatic schedule, but the test runs based on the last time it ran and not a time you select. Since we want these tests to run in the middle of the night, it would be nice to be able to select that.

Also, it would be nice to be able to generate API documentation.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

No issues with stability.

What do I think about the scalability of the solution?

No issues with scalability.

How are customer service and technical support?

Technical support is fantastic.

Which solution did I use previously and why did I switch?

We used Postman. Runscope makes it much easier for those without a programming background to accomplish API testing.

How was the initial setup?

I found setup to be pretty straightforward and anything that gave me trouble was solved by looking at the Runscope documentation.

What's my experience with pricing, setup cost, and licensing?

Pricing is fair.

What other advice do I have?

It's really easy to use and the documentation is extremely helpful if you get stuck on something. The product is flexible and allows you to set up variables and assertions in order to customize tests for your needs.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
BM
DevOps Engineer at a tech company with 1-10 employees
Real User
Integration with alerting apps tightens our feedback loop during testing

Pros and Cons

  • "This solution has improved our responsiveness to issues in both production and non-production environments. Through integration with Slack, our development and DevOps teams are quickly aware of issues related to new changes deployed to non-production environments, which tightens our feedback loop during testing. Through integration with PagerDuty, our DevOps resources are able to reduce the adverse impact of a given production issue."
  • "Tests are per-step configurable with script capabilities and support for token-based authentication using variables. Configurable test frequency and integration with both Slack and PagerDuty for alerting our DevOps team are also important. General use of the dashboard by multiple team members is also valuable."
  • "We would like the ability to configure Runscope behavior when a given test has its schedule modified. That is, in a non-production environment, we often need to pause a test (either by removing it from the schedule or adjusting its frequency) and this results in the automatic kick-off of this test in all other environments. We would like the ability to simply pause a test for a specific shared environment without causing additional tests to be started."
  • "We definitely make use of historical test results, especially when tracking more difficult issues such as timeouts. Per our test frequency (15 minutes) we often need to go back further than the last 100 test runs. We are aware that results further back in time are available through the Runscope API, however, it would be great if the dashboard allowed easy access to a larger amount of historical data."

What is our primary use case?

API monitoring for IoT data pipeline in the cloud.

How has it helped my organization?

This solution has improved our responsiveness to issues in both production and non-production environments.

Through integration with Slack, our development and DevOps teams are quickly aware of issues related to new changes deployed to non-production environments, which tightens our feedback loop during testing. Through integration with PagerDuty, our DevOps resources are able to reduce the adverse impact of a given production issue.

Runscope has also helped to expose subtle network latency or routing issues with components managed by our cloud provider.

What is most valuable?

API monitoring where specific tests are designed to emulate production data-flows through our pipelines. Tests are per-step configurable with script capabilities and support for token-based authentication using variables. Configurable test frequency and integration with both Slack and PagerDuty for alerting our DevOps team are also important. General use of the dashboard by multiple team members is also valuable.

What needs improvement?

  1. Ability to configure Runscope behavior when a given test has its schedule modified. That is, in a non-production environment, we often need to pause a test (either by removing it from the schedule or adjusting its frequency) and this results in the automatic kick-off of this test in all other environments. We would like the ability to simply pause a test for a specific shared environment without causing additional tests to be started.
  2. We definitely make use of historical test results, especially when tracking more difficult issues such as timeouts. Per our test frequency (15 minutes) we often need to go back further than the last 100 test runs. We are aware that results further back in time are available through the Runscope API, however, it would be great if the dashboard allowed easy access to a larger amount of historical data.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

There have been some Runscope outages in the last year. Though not many, Runscope availability is of critical importance in our production context. This was the primary reason for the "recommend this solution" score of eight rather than nine or 10.

What do I think about the scalability of the solution?

No issues with scalability.

How is customer service and technical support?

Tech support is responsive through website chat (this is the only communication channel we have used).

How was the initial setup?

Setup was fairly straightforward.

What's my experience with pricing, setup cost, and licensing?

Having the different plans organized by team size, buckets, and requests per billing period works for our needs. Larger companies may need to take a close look at these elements.

Which other solutions did I evaluate?

Some alternatives were evaluated prior to choosing Runscope, however, I do not have access to that information at this time.

What other advice do I have?

Create a DevOps team story to evaluate Runscope, which includes setup for all tests/buckets which will be used in your production environment. Exercise all use cases and outcomes, including integration with your desired tools (e.g. Slack, PagerDuty, etc.), and tune the available configurations as needed.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
SB
Director of Engineering at a tech services company with 11-50 employees
Real User
Helps us verify that production deployments do not break the features our customers use

Pros and Cons

  • "It helps us verify that production deployments do not break the features our customers use."
  • "We have the ability to populate environment variables using scripts. This allows us to keep the configuration mostly dynamic and avoid a lot of manual updates when something changes."
  • "The initial setup included a lot of repetitive manual work."
  • "The user interface requires a lot of clicking around."
  • "We would like to have an improved ability to share a set of tests across different environments (e.g., dev, stage, and prod)."

What is our primary use case?

We are using it for API end-to-end testing.

How has it helped my organization?

It helps us verify that production deployments do not break the features our customers use.

What is most valuable?

We have the ability to populate environment variables using scripts. This allows us to keep the configuration mostly dynamic and avoid a lot of manual updates when something changes. 

Schedules are a great way to have repetitive checks against the environment.

What needs improvement?

We would like to have an improved ability to share a set of tests across different environments (e.g., dev, stage, and prod). The shared environments feature is great in practice, but in reality, if you have two or three of them using the same set of tests, it gets complicated to figure out which environment is failing. 

Overall, the user interface (UI) requires a lot of clicking around. 

Waiting for asynchronous results requires a lot of "Pause" steps, which is quite difficult to get right. It would be great to have some type of "Waiting for Results" step, which would include success/failure conditions, then eventual timeout for retries.

For how long have I used the solution?

More than five years.

What do I think about the stability of the solution?

There have been no stability issues.

What do I think about the scalability of the solution?

There have been no scalability issues.

How are customer service and technical support?

We have found working with the technical support to be great.

Which solution did I use previously and why did I switch?

We did not previously use another solution.

How was the initial setup?

The initial setup was straightforward, but it included a lot of repetitive manual work.

What's my experience with pricing, setup cost, and licensing?

Not applicable.

Which other solutions did I evaluate?

We evaluated AppNeta, New Relic Synthetics, and custom implementations.

What other advice do I have?

Create a script to populate tests from a config file. Store it in a source control and load/update it through the API. Doing things through the UI wastes a lot of time.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
KF
QA Manager at a retailer
Real User
Environment initial variables allow our team to run tests against multiple environments on the fly

Pros and Cons

  • "Environment initial variables allow our team to run tests against multiple environments on the fly."
  • "We utilize modular tests based on environment variables. These have saved QA a lot of time when setting up our back-end automation."
  • "If you forget to set the default environment during test creation, you have to go through every test and make sure that it is set to the environment that you prefer if using shared environments. If there was an option to force all tests within a bucket into a specific environment with one click, this would save a lot of time."

What is our primary use case?

Our primary use is for REST API automation.

How has it helped my organization?

Runscope allows our QA team to spend less time on our API smoke test during releases.

What is most valuable?

Environment initial variables: These allow our team to run tests against multiple environments on the fly. 

We utilize modular tests based on environment variables. These have saved QA a lot of time when setting up our back-end automation.

What needs improvement?

One area that I seem to forget adjusting when creating new tests is setting the default environment within test settings. If you forget to set the default environment during test creation, you have to go through every test and make sure that it is set to the environment that you prefer if using shared environments. If there was an option to force all tests within a bucket into a specific environment with one click, this would save a lot of time.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

Stability has been fantastic.

What do I think about the scalability of the solution?

Not so far.

How are customer service and technical support?

Technical support has been amazing. I have reached out with chat many times, and I have received help immediately every time.

Which solution did I use previously and why did I switch?

We did not previously use another solution.

How was the initial setup?

The initial setup was straightforward. Now that I have more experience with REST API automation, anything that I ran into during setup was due to my lack of knowledge at the time.

What's my experience with pricing, setup cost, and licensing?

Runscope pricing is unbeatable.

Which other solutions did I evaluate?

I did not evaluate another solution. I had a lot of recommendations that Runscope was the tool to use.

What other advice do I have?

Runscope is the best option for REST API automation.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
RL
Project Manager, QA Leader and Information Security Manager at a tech services company with 11-50 employees
Real User
Notifications enable us to proactively fix issues before the customer even notices them

Pros and Cons

  • "As we have the APIs scheduled, we use the notifications. We know the exact moment it is breaking something that might impact the customer system, that might result in a feature not working. We know "in advance" and we can fix it before the customer notices. It helps us to be more proactive."
  • "When it's a problem on our side, the server is down, or there is a lack of memory, the first thing to break is the API, because the calls can't be processed. So we know exactly at that moment because we receive notifications, and we can take action. It doesn't only notify us of the type problem, but it also links it to our servers."
  • "The schedule is definitely the most valuable. Also, being able to check the JSON attribute values is helpful. We always check the attributes. Finally, the notifications are helpful as well."
  • "One thing that can be improved is the logging. Sometimes, when we have an error, we don't know how to find more information about the problem that has happened. Sometimes, it just returns something like "that gateway." More information would be a way to improve it."
  • "Currently, we can only schedule to the nearest five minutes, one hour, two hours, etc. If we could set up the exact time that we want it to run, that would be good because sometimes we have one service that crosses another service."

What is our primary use case?

We use it to monitor the APIs. We always have them scheduled there.

How has it helped my organization?

As we have them scheduled, we use the notifications. We know the exact moment it is breaking something that might impact the customer system, that might result in a feature not working. We know "in advance" and we can fix it before the customer notices. It helps us to be more proactive.

We have it all monitored. Even when it's a problem on our side, the server is down, or there is a lack of memory, the first thing to break is the API, because the calls can't be processed. So we know exactly at that moment because we receive notifications, and we can take action. It doesn't only notify us of the type problem, but it also links it to our servers.

What is most valuable?

The schedule is definitely the most valuable. Also, being able to check the JSON attribute values is helpful. We always check the attributes. Finally, the notifications are helpful as well.

What needs improvement?

One thing that can be improved is the logging. Sometimes, when we have an error, we don't know how to find more information about the problem that has happened. Sometimes, it just returns something like "that gateway." More information would be a way to improve it.

Also, currently we can only schedule to the nearest five minutes, one hour, two hours, etc. If we could set up the exact time that we want it to run, that would be good because sometimes we have one service that crosses another service. It's rare that this happens, but sometimes it does happen because the service loads are loading at the same time. What our tests were doing is creation, addition, and deletion. Before we deleted, this one service crossed the results of another one. And then we deleted the thing and broke another service. It was just a coincidence of the same millisecond, so one thing that could solve this would be if we could schedule the exact time that we want, or to schedule the order of service execution. That would be a good improvement.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

No issues with stability.

What do I think about the scalability of the solution?

No issues with scalability. It's fine.

How are customer service and technical support?

Tech support has been pretty good. We had a problem. We accidentally deleted a bucket. We were trying to create a script and deleted a bucket. I needed to ask support to use the backup to get things up. Maybe a good feature would be to show this backup inside the tool, so end-users can just do backups by themselves.

Which solution did I use previously and why did I switch?

We did use Postman but not for scheduling; just to do a quick check, we ran it from there. I don't know too many tools in the market, so it's hard to make a comparison. I think Postman does notifications and schedules stuff, but I don't know the price, so I don't know if it would be similar or totally different.

How was the initial setup?

The setup was very nice. It was fine.

What's my experience with pricing, setup cost, and licensing?

I think it's a good price. Once I asked my boss to approve it, he approved it immediately.

What other advice do I have?

I would rate Runscope an eight out of 10 because it's still missing flexible scheduling, the exact time or the ability to sort the services the way we want them to run.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
it_user884064
Quality Assurance Engineer at a consultancy with 1,001-5,000 employees
Real User
It has the ability to run a large amount of tests simultaneously

Pros and Cons

  • "It is a relatively reliable tool for running big amounts of API tests, which you can schedule to run periodically, as the product is very good."
  • "It has a relatively simple user experience."
  • "We can receive an email and slack notifications if tests are failing/passing."
  • "There are not many options for scalability."
  • "Needs the ability to create code which runs before and after API tests."
  • "If triggering test bucket via trigger API, there is no option to see if the tests which were triggered passed or failed."

What is our primary use case?

Our primary use case for this solution is API tests.

How has it helped my organization?

Since we are working with Runscope, we have a tool which is shareable between all the team members in a uniform standard. Now, each of us can easily run all our tests in a couple of minutes before deployment and all the team can see the results. We can receive an email and slack notifications if tests are failing/passing.

What is most valuable?

  • Ability to schedule and compare the test suite results over time periods. 
  • It has a relatively simple user experience. 
  • Ability to copy tests suite between buckets. 
  • Ability to add JS libraries. 
  • Email and slack notifications. 
  • Ability to run a large amount of tests simultaneously. 
  • Revision option (to track changes). 
  • Shared environment for tests.

What needs improvement?

  • Ability to create code which runs before and after API tests. 
  • The area of "initial script" is relatively small and incomplete related to code editors. 
  • There are no regular features and validations for editor. 
  • If triggering test bucket via trigger API, there is no option to see if the tests which were triggered passed or failed.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

During the using period, we experienced several downtimes.

What do I think about the scalability of the solution?

There are not many options for scalability.

How are customer service and technical support?

The support is very good. They answering quickly and effectively.

Which solution did I use previously and why did I switch?

We were using editors to create scripts and run them locally. This was unstable, not shareable, or a uniform solution. We were not able to run big amount of tests simultaneously and receive notifications.

How was the initial setup?

The initial setup was pretty straightforward. The work is in the cloud, you have to set integration options and libraries, if needed, then start to work.

What's my experience with pricing, setup cost, and licensing?

The price for the product is very reasonable for a team of more than four to five members.

Which other solutions did I evaluate?

We did evaluate other solutions before choosing Runscope.

What other advice do I have?

It is a relatively reliable tool for running big amounts of API tests, which you can schedule to run periodically, as the product is very good.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
NM
Test Engineer
Real User
Detects API layer issues in our production environment before our customers do

Pros and Cons

  • "The feature of automatically executing tests at a predefined time interval is great as we can be continuously testing our end-points before and after builds are deployed."
  • "Reliability. I would like to see more money invested in disaster recovery testing of the application. The product should be hosted across several AWS regions so if there are issues with an EC2 instance in a particular region, the whole application won't be affected. Once or twice a year there are issues with AWS that causes the reliability of the application to suffer."

What is our primary use case?

API monitoring and testing.

How has it helped my organization?

Runscope is able to detect API layer issues in our production environment before our customers are able to.

What is most valuable?

Automatic scheduling of test execution. The feature of automatically executing tests at a predefined time interval is great as we can be continuously testing our end-points before and after builds are deployed.

What needs improvement?

Reliability. I would like to see more money invested in disaster recovery testing of the application. The product should be hosted across several AWS regions so if there are issues with an EC2 instance in a particular region, the whole application won't be affected.

For how long have I used the solution?

One to three years.

What do I think about the stability of the solution?

Once or twice a year there are issues with AWS that causes the reliability of the application to suffer.

What do I think about the scalability of the solution?

No issues with scalability.

How are customer service and technical support?

Tech support was more responsive prior to the acquisition by CA Technologies.

Which solution did I use previously and why did I switch?

Previously used Postman. I needed the scheduling feature that Postman, at the time, did not have.

How was the initial setup?

Setup was moderately straightforward.

What other advice do I have?

Try it out and see if this product will work for you.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
Product Categories
API Monitoring Software
Quick Links