What is most valuable?
The charts dashboard, especially the results screen, is the most
valuable feature. I’ve also configured the tests, so I’ve also used the
test definition page, which is very easy to use, self-explanatory, and
simple. The real value comes from the results screen.
How has it helped my organization?
StormRunner Load helped us evaluate the performance of the product version we were about to launch. It showed us bottlenecks, where our product reaches its limits, and different characteristics that affect this. For example, we had metrics we thought were affecting our response time. Initially we thought there were two metrics, but we found another significant metric we should have taken into account. Without StormRunner Load we wouldn’t have known about this "missing" metric.
What needs improvement?
I think the report page could be improved, although we have used the reporting feature. It wasn’t accurate enough to use as a report that I could send my managers. I had to take additional screenshots to help fill in the reporting. I actually, ultimately, generated a report on my own.
For how long have I used the solution?
I've been using it for two months.
What was my experience with deployment of the solution?
What do I think about the stability of the solution?
What do I think about the scalability of the solution?
Which solution did I use previously and why did I switch?
There was no previous solution in place.
What was our ROI?
This solution immediately helped us to determine the priorities for what to develop next and where we would reach our limits. Because we had a clear prediction, we knew when it would be urgent to develop features, in various cases. Before StormRunner Load, we didn’t have a clear developer roadmap, and now its very clear where we should invest, our weakest links, and what we need to do to strengthen weak links.
If you’re using third-parties for networking, cloud based servers, load balancer, VPN providers etc., all of them should be informed about the expected load. We had a VPN provider that was expecting thirty users per hour and calibrated accordingly, and the minute we started our load test it shut down. This third-party was not ready. Once we informed them that we were increasing our load, they increased their ability to support the load test.
Which other solutions did I evaluate?
No, this was my first choice.
What other advice do I have?
It is very cost effective to start with a low number of virtual users for a short amount of time in order to verify that your company system is running properly. Start with a small load test, run it until completion, verify that you see the logs you intended to see, and all people involved in the environment are ready to see what they expected.
Disclosure: My company has a business relationship with this vendor other than being a customer: I used to work for HP.
Jul 14 2015