What is most valuable?
- Recording and playback
- Load testing
How has it helped my organization?
Our benchmark load tests for a major release involving various scenarios with a combination of different virtual users relied on the recording, playback and eventual load testing capabilities of OATS.
What needs improvement?
- The tool runs out of memory when the recording flows are long and involve many screens despite increasing the memory settings for OpenScript
- We found the tool impossible to use for client side performance testing without filtering out JS, CSS, PNG etc as the method would error due to a Java method size limitation.
- Parameterization of variables using regular expressions was not a foolproof way of running tests as it would mostly work for a single user but fail during load testing.
- Parameterization involving SQL queries was not straightforward
- Except a YouTube video/PDF which covers some basics and is probably outdated now, there are no tutorials available for other functionalities that the recording tool provides.There is no in-depth tutorial of the load testing interface and troubleshooting steps.
- Collection of system metrics - although there are multiple parameters defined, it did provide a holistic picture of the state of the system.We had to implement our own way of obtaining this.
- Load testing web interface did not provide details about errors. The only way to correlate was to inspect server logs.
For how long have I used the solution?
What was my experience with deployment of the solution?
What do I think about the stability of the solution?
Yes, 12.2 was not stable. Upgrading to 12.4.x resolved a lot of previously known issues
What do I think about the scalability of the solution?
Which solution did I use previously and why did I switch?
This tool did a much better job recording the http requests during web load testing than other open source tools. Cookie management and other url parameters were handled efficiently and were easier to identify in the tree view.
How was the initial setup?
It was straightforward for the most part except that at times the services wouldn't run and un-installation messed things up.
What about the implementation team?
What was our ROI?
I believe we made a wise choice despite some shortcomings and hiccups.
Which other solutions did I evaluate?
We evaluated other options.
What other advice do I have?
Read the tutorial and try out a few scenarios before going full throttle.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Jan 14 2015