What is most valuable?
The breadth of protocol support means that you can get LoadRunner (and its sister product Performance Center) to test almost any application with useful built-in functions. The programmable scripts allow for the creation of sophisticated reliable and robust frameworks and scripts. The API potentially allows you to extend the tool to support applications that it cannot support, but how to do this is not well documented or supported.
The analysis capability is simply best of breed as it allows you to take data from any source and correlate it with your results. In other tools, analysis is more problematic and therefore a bit of a black art.
How has it helped my organization?
We use LoadRunner and its sister product Performance Center as two of the key pieces in our corporate tool kit along with other tools in delivering performance engineering services. Some of our customers insist on using this, and for them, it has helped administer payroll, deliver luggage in airports, and enabled them to support tax returns.
At Infuse, we have integrated such tools very early into early performance validation at the unit test level. This enables our customers to find defects early on in development, and combined with performance modelling in design, enables them to fix performance issues early and cheaply.
What needs improvement?
HP as a company has been slow to move to support modern technologies e.g. Oracle ADF. HP took a long time to support the protocol. The support could be better.
For how long have I used the solution?
I have, as a performance engineer, been using LoadRunner since 1999, in multiple different versions. As a business we have been using it since 2002, and it features in around 80% of our performance engineering assignments along with other tools.
What was my experience with deployment of the solution?
The tool itself is very easy to set up and install the various components. 90% of the time there are no deployment issues. The biggest issue really, is setting up the connections from the load generators to the controller component. However, these are are normally network related issues that, with the correct input from the network/infrastructure team, can quickly be resolved.
What do I think about the stability of the solution?
No, the application is very robust. The only time issues are experienced is when it is running for extra-long period soak tests, e.g. for 24 hours. Then, only on odd occasions, does the system sometimes experience issues with results files, which become corrupt. Usually this is because of issues during the test with the AUT causing very large results files being generated.
What do I think about the scalability of the solution?
No, the application is scalable. Additional load testing capacity can be added quickly and easily by adding another load generator. Also, some GUI based scripting protocols such as TruClient do utilize large amounts of memory and CPU power, which needs to be planned for carefully if running tests over long periods with lots of users.
How are customer service and technical support?
Average, to occasionally good, but no better. Technical Support
HP LoadRunner technical support is adequate, with implementation issues and functional problems that may arise. However, it doesn’t include scripting support, but there is a wealth of information on numerous public forums regarding LoadRunner to assist most technical issues which helps.
Which solution did I use previously and why did I switch?
As an independent vendor we use JMeter, The Grinder and Oracle Application Tool Suite’s load test module, but all of these tools don’t have the breadth of supporting protocols and so are not always suitable for our client’s applications and systems being tested. They are also sub-optimal from an analysis perspective, but do a good job.
How was the initial setup?
The setup for LoadRunner is straightforward as the installations are via wizards, and this makes them quick and easy. Make sure though, that the instructions are followed as written, and be aware that virus detection software can adversely affect the installation. The most complex area is in ensuring connectivity between the load generators and the controller, which may involve firewall changes as load generator machines should be located where the users are situated or externally on the cloud. The complexity is often getting permissions from the IT support teams and explaining what you are doing rather than changing the firewall itself and setting it up. It requires good communication skills.
What about the implementation team?
We implement it ourselves.
What was our ROI?
Considering the performance issues that we have found and prevented moving into production on client sites is enough to show that there is a positive ROI with the tool. This is despite the initial expense, as assets produced can be re-used on each implementation reducing the cost for subsequent releases.
What's my experience with pricing, setup cost, and licensing?
The set up costs depend on the protocol type (there are standard and premium types), the number of virtual users being simulated, the terms, and the number of, load generator machines that are needed. Then there is an annual support maintenance fee that allows for free upgrade and access to the support functions.
Which other solutions did I evaluate?
We would evaluate all the tools on the market for a customer before making a recommendation. We do an evaluation based on the customers dynamics/culture, budget, and delivery methodology, and then offer our client a choice.
Other tools though don’t provide the breadth of support for different protocols and so this tool is limited to the applications it compatible with. It also has more basic reporting and analysis features. But then other tools are also generally cheaper.
What other advice do I have?
it is a best of breed product with some minor frustrations around product support.