What is most valuable?
A while ago, we were looking for a tool to do testing across the web services. During that time, we were doing some PoCs on SoapUI. We did a PoC with Lisa. The company name was ITKO. After going through several PoC sessions, we figured out that this would be beneficial because the functional testing, which we would be creating for those web services, could be leveraged for performance testing as well.
We have a conceptual testing document where we say, you run a single user-firing transaction against three simulators and you host your test there. Over a period of time, we matured from using web services to using all the web-based UI applications. The best part about it is that it goes one layer below the UI. It’s not dependent on the UI. It’s dependent on the transactions going back and forth between the application server and one layer below your browser.
Usually, with tools like Selenium, you just click. In this case, you post a transaction and test how far it goes.
How has it helped my organization?
What needs improvement?
At the point of next release, we would really like to see the issues being resolved which we are facing right now with dynamic applications. Now, most of the applications in the market, in the industry, and most of the companies are rolling out responsive apps. If you can get rid of that plugin interface, you can get straight into the application itself, that would really work for us.
What do I think about the stability of the solution?
Initially, there were challenges. It has a heavy client. The rendering of the application itself and the usage of certain features was unclear and we had to ask them how to use them. Improvements to the documentation were made over a period of time. So, right now, other than the fact that it doesn’t work for a few applications, it’s pretty much stable right now. It is maturing. Pretty much it's stable enough right now where we don't have to worry about stability.
What do I think about the scalability of the solution?
In terms of individual usage, it is a one-user, one-license model. On the performance side of the usage, it is pretty much decent enough where we are able to put in low levels of total users and again with the feature of authorizing IPs.
Technically, you can scale it up to a point where you have your IP stability. So, scalability-wise, from the performance aspect of it, we never have any complaints on that. We just needed to have the infrastructure to support it.
How are customer service and technical support?
We used to use technical support quite often. It has lowered down because the fact that we now have in-house expertise. So, we are working as mentors for people on the usage part of it.
Of course, when it comes to tool usage and if something breaks on the tool side, we have our in-house team take a look at it first. They are the software engineering group. Then, if nothing happens on that side, we go back to customer service and say, “hey technical guys.” We contact the technical support.
Which solution did I use previously and why did I switch?
We were using TestPartner, which is a UI-based application for capturing and creating regression test cases for regression test automation. It was VB based and at the same time you couldn't scale it up. You could only leverage the same functional test for performance. So, the advantage we have with CA Service Virtualization is that whenever you create any function regression, we can use it, as is, for performance testing. We host it out there on the simulator, saying that, now, instead of running one user, run 500 users. Good to go.
But, your PC needed to be on when TestPartner ran. You could not lock your machine because it was all keyboard and mouse driven. We didn't want that.
How was the initial setup?
Initial setup was understandable. It wasn't that complex. The complexity was setting up the infrastructure. That was one of the challenging things where we had to acquire a few machines so we could set up the coordinators and simulators. We wanted beefier machines so we could get very similar production load. But, other than that, getting it on the users machine is a straightforward process.
Which other solutions did I evaluate?
We were looking at performance testing tools. We had HPE LoadPunner before. We looked at it, but that was a while ago. But, with the benefits of service installation that Lisa provided, along with a performance background with the infrastructure for supporting performance testing as well, at that point in time, it was a one-tool-fits-all application for us.
What other advice do I have?
This is for those who are looking for tools where they would really like to leverage the service installation piece. One tool doing service installation. One tool carrying out performance testing. One tool carrying out functional testing. And if they want to not write performance testing specifically and instead leverage the functional test, I think this would be a tool to be considered.