StormRunner Load Review
Enables developers and testers to quickly automate the user actions at the browser level, which also helps measuring the end user experience by measuring the browser rendering times.


Valuable Features

StormRunner Load reduces the overall time for performance testing engagement and is best suited for the products in agile process. Easy to automate and execute the tests, StormRunner Load provides a feature to compare the current test results with the previous results to see if the current build is better than the previous or not.

With the ability to integrate with monitoring tools like SiteScope and APM tools like New Relic, performance bottlenecks in the applications can be identified very quickly and easily.

Protocol Support

StormRunner Load supports Web, Web Services, SAP Web and JMeter scripts

Easy Automation

StormRunner Load uses Truclient and TruAPI (which uses node.js) which enables developers and testers to quickly automate the user actions at the browser level, which also helps measuring the end user experience by measuring the browser rendering times as well. Performance testers can upload their LoadRunner scripts that are developed using LR 12.XX to StormRunner Load directly.

High Scalability

StormRunner Load is highly scalable and supports upto 1 million concurrent users.

On Premise Load Generators

StormRunner Load introduces the concept of On Premise Load Generators to compare the application performance within the network to various other geographical locations.

Configure SLAs

SLAs can be configured in StormRunner Load during the test execution. The tool highlights all the transactions that did not meet the SLAs and further drill down can help the users understand the geographical locations and times where the SLAs are not met.

Real time anomaly detection

During the test execution, StormRunner Load detects any anomalies where the test results are deviated for a specific transaction / geographical location / script / counter to identify the issues real time and further drill down provides more details about the anomalies.

Integration with Monitoring Tools

StormRunner Load integrates with monitoring tools like SiteScope to provide the hardware and application level metrics during the performance tests. StormRunner Load can also integrate with New Relic and other similar APM tools that helps identifying the server side bottlenecks so that organization can reuse investments

Improvements to My Organization

HP StormRunner Load has provided us a) flexibility for on demand load generation b) reduced over all TCO on our load testing efforts c) Reduced the skill level to develop load test scripts.

Room for Improvement

Custom load patterns for the performance tests: Currently StormRunner Load supports only linear ramp up of the user load and doesn’t provide the user load to follow a step incremental load pattern or any other pattern that a performance tester wants to.

Load distribution among on premise LGs: StormRunner Load currently supports load generation from only one of the On premise load generators and the user load cannot be distributed to multiple on premise LGs.

Simulation of latencies: There is no provision to simulate network latencies to simulate realistic geographical locations and mobile networks.

Ability to create custom reports: StormRunner Load provides only the default reports and there is no provision available currently to generate the custom reports

Backward compatibility: StormRunner Load expects the LoadRunner scripts that are generated from LR 12.00 and above. Any scripts developed using the older versions are not supported.The work around to overcome this problem is to open the scripts in LR 12.00 or higher and save it again before uploading to StormRunner Load..

Use of Solution

We have used the product for more than 14 weeks now and tested on some of our internal applications in the organization that would require performance testing.

Deployment Issues

HP StormRunner Load doesn’t need any installation or deployment unless you wanted to configure few on premise load generators. Configuring on premise load generators is pretty straight forward. All that you need is to have an account with HP SaaS to start using StormRunner Load.

Stability Issues

HP SaaS web application is pretty stable and no issues were seen during the evaluation process.

Scalability Issues

There were no issues with the scalability, however, the time taken to initialize the test is directly dependent on the number of users to be simulated.

Customer Service and Technical Support

Customer Service:

5/5.

Technical Support:

The support provided to us very good which helped us to evaluate the solution for our context.

Previous Solutions

We are in the process of adopting a SaaS based option, this led us to evaluate a bunch of tools.

Initial Setup

It is simple - just registering with HP SaaS platform called pronq.com.

Implementation Team

We evaluated through our in-house resources.

ROI

Our estimated ROI is positive we need a couple of months to gain insight into actual ROI.

Pricing, Setup Cost and Licensing

HP StormRunner load is SAAS offerings so the initial setup costs are very minimal when compared to the traditional approach (no or minimal capex ), and the day to day costs would be purely driven by the duration and number of virtual users required to simulate the traffic.

Other Solutions Considered

We have evaluated HP StormRunner Load against other market competitors that include Blazemeter, SOASTA’s Cloud Test, NeoLOAD, VSTS, Load Complete and StressStimulus.

Other Advice

HPStormRunner load is an apt product a) if your application stack has Web 2.0 traffic b) your requirement is for a service that allows for pay as you go model c) proven and reliable load testing solution with superior support.

This review was co-authored by Sravanthi Naga & Ramanjaneyulu Narra, Performance Architects at Gallop Solutions. Gallop is a US based Co-located Independent Testing Services operating since 2003 with offices in Philadelphia & California. World's largest and leading organizations have relied on Gallop's specialist independent software testing services for more than a decade and have achieved significant market acceleration, returns on investments in their software quality initiatives.

Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
9 visitors found this review helpful

4 Comments

it_user232089Real User

Helpful review.

30 April 15
it_user232107Real User

Good review to understand the Pros N Cons

30 April 15
Subhendu PattnaikReal UserTOP 20

Thank you Prakash and Ed.

01 May 15
it_user232266Vendor

Hey,
Thanks for the detailed review :)

A couple of comments from StormRunner R&D:

1. We can scale higher than a Million Vusers. We tested for 2 Million, and we are looking for customers who need to scale higher so we can validate our capabilities.
2. Our latest release, delivered on April 26th, includes an integration with HP Network Virtualization (formerly known as Shunra). This integration allows emulating of network conditions, including Latency and Packet Loss.
3. When on-premise LGs are used, currently the load is distributed equally on all LGs allocated to the test.

We appreciate you taking the time to write this, and we plan to follow your comments with regards to Room For Improvement.

Thanks and regards,

Oded Keret
StormRunner R&D

01 May 15
Guest
Why do you like it?

Sign Up with Email