What is most valuable?
The facilities for testing API’s (services, REST API’s, messaging clients, etc..). Especially the features for parametrizing requests and checking responses are very powerful. But the product really shines in end-to-end testing, not only because of all the built-in tools supporting this, but also because of the great extensibility of the tool.
If you want something that’s not provided out of the box, then you can write it yourself and integrate it with SOAtest.
As an expert user, you can accomplish a lot with this tool.
How has it helped my organization?
Using this tool, we were able to set up a nightly, still growing, company-wide regression test -- dramatically decreasing time being spent on (manual) regression testing.
What needs improvement?
- Web-testing facilities. This was not very good when we were using version 9.2, so we built our own Selenium framework and integrated this with SOAtest (this being possible at all is a very positive point in my opinion ;-).
- The user-interface is not very lightweight or friendly, sometimes buggy and cumbersome. The product has a steep learning curve, but very well worth the effort.
- There is virtually no (online) user community, except for an outdated forum. When having issues you have to find everything out yourself (or call in help from Parasoft).
- Reporting facilities can be better.
- Last but not least: using the tool in current continuous integration/delivery practices is not very easy or straightforward.
For how long have I used the solution?
What do I think about the stability of the solution?
Yes, the product can be buggy or crash sometimes. But not to the extent that it hinders working with it.
What do I think about the scalability of the solution?
We had some issues with stability and performance -- with a growing testbase and/or ‘big’ tests. But these also were minor issues.
How are customer service and technical support?
8 out of 10.
Technical staff living nearby are always prepared to come by and help with issues. Communicating with developers abroad is also satisfactory.
Which solution did I use previously and why did I switch?
Not really, QTP was used a little before but was not the right tool for our purposes.
How was the initial setup?
It was complex, but that had more to do with our own complex infrastructure than with the product itself. But it took some time to get the setup of the product right.
What's my experience with pricing, setup cost, and licensing?
It is an expensive product, so think carefully about whether it fits your purposes and is the right tool for you. Support pack is helpful. Due to high license costs, the tool is not easy to use by many people in your organization (like with open-source tools) and so to make test automation something for the whole team.
Which other solutions did I evaluate?
Yes, but before I was involved, so I don’t know more than that.
What other advice do I have?
Take your time to get to know SOAtest very well, don’t get put off by the steep learning curve, it’s very well worth the effort.