What is most valuable?
StormRunner Load reduces the overall time for performance testing engagement and is best suited for the products in agile process. Easy to automate and execute the tests, StormRunner Load provides a feature to compare the current test results with the previous results to see if the current build is better than the previous or not.
With the ability to integrate with monitoring tools like SiteScope and APM tools like New Relic, performance bottlenecks in the applications can be identified very quickly and easily.
StormRunner Load supports Web, Web Services, SAP Web and JMeter scripts
StormRunner Load uses Truclient and TruAPI (which uses node.js) which enables developers and testers to quickly automate the user actions at the browser level, which also helps measuring the end user experience by measuring the browser rendering times as well. Performance testers can upload their LoadRunner scripts that are developed using LR 12.XX to StormRunner Load directly.
StormRunner Load is highly scalable and supports upto 1 million concurrent users.
On Premise Load Generators
StormRunner Load introduces the concept of On Premise Load Generators to compare the application performance within the network to various other geographical locations.
SLAs can be configured in StormRunner Load during the test execution. The tool highlights all the transactions that did not meet the SLAs and further drill down can help the users understand the geographical locations and times where the SLAs are not met.
Real time anomaly detection
During the test execution, StormRunner Load detects any anomalies where the test results are deviated for a specific transaction / geographical location / script / counter to identify the issues real time and further drill down provides more details about the anomalies.
Integration with Monitoring Tools
StormRunner Load integrates with monitoring tools like SiteScope to provide the hardware and application level metrics during the performance tests. StormRunner Load can also integrate with New Relic and other similar APM tools that helps identifying the server side bottlenecks so that organization can reuse investments
How has it helped my organization?
HP StormRunner Load has provided us a) flexibility for on demand load generation b) reduced over all TCO on our load testing efforts c) Reduced the skill level to develop load test scripts.
What needs improvement?
Custom load patterns for the performance tests: Currently StormRunner Load supports only linear ramp up of the user load and doesn’t provide the user load to follow a step incremental load pattern or any other pattern that a performance tester wants to.
Load distribution among on premise LGs: StormRunner Load currently supports load generation from only one of the On premise load generators and the user load cannot be distributed to multiple on premise LGs.
Simulation of latencies: There is no provision to simulate network latencies to simulate realistic geographical locations and mobile networks.
Ability to create custom reports: StormRunner Load provides only the default reports and there is no provision available currently to generate the custom reports
Backward compatibility: StormRunner Load expects the LoadRunner scripts that are generated from LR 12.00 and above. Any scripts developed using the older versions are not supported.The work around to overcome this problem is to open the scripts in LR 12.00 or higher and save it again before uploading to StormRunner Load..
For how long have I used the solution?
We have used the product for more than 14 weeks now and tested on some of our internal applications in the organization that would require performance testing.
What was my experience with deployment of the solution?
HP StormRunner Load doesn’t need any installation or deployment unless you wanted to configure few on premise load generators. Configuring on premise load generators is pretty straight forward. All that you need is to have an account with HP SaaS to start using StormRunner Load.
What do I think about the stability of the solution?
HP SaaS web application is pretty stable and no issues were seen during the evaluation process.
What do I think about the scalability of the solution?
There were no issues with the scalability, however, the time taken to initialize the test is directly dependent on the number of users to be simulated.
How are customer service and technical support?
5/5. Technical Support
The support provided to us very good which helped us to evaluate the solution for our context.
Which solution did I use previously and why did I switch?
We are in the process of adopting a SaaS based option, this led us to evaluate a bunch of tools.
How was the initial setup?
It is simple - just registering with HP SaaS platform called pronq.com.
What about the implementation team?
We evaluated through our in-house resources.
What was our ROI?
Our estimated ROI is positive we need a couple of months to gain insight into actual ROI.
What's my experience with pricing, setup cost, and licensing?
HP StormRunner load is SAAS offerings so the initial setup costs are very minimal when compared to the traditional approach (no or minimal capex ), and the day to day costs would be purely driven by the duration and number of virtual users required to simulate the traffic.
Which other solutions did I evaluate?
We have evaluated HP StormRunner Load against other market competitors that include Blazemeter, SOASTA’s Cloud Test, NeoLOAD, VSTS, Load Complete and StressStimulus.
What other advice do I have?
HPStormRunner load is an apt product a) if your application stack has Web 2.0 traffic b) your requirement is for a service that allows for pay as you go model c) proven and reliable load testing solution with superior support.
This review was co-authored by Sravanthi Naga & Ramanjaneyulu Narra, Performance Architects at Gallop Solutions. Gallop is a US based Co-located Independent Testing Services operating since 2003 with offices in Philadelphia & California. World's largest and leading organizations have relied on Gallop's specialist independent software testing services for more than a decade and have achieved significant market acceleration, returns on investments in their software quality initiatives.