What is most valuable?
SOAtest is, at its base, a collection of tools that can be combined to generate test scenarios.
Input data can be gathered from any possible source from web GUIs, databases, Excel, and files and is used in SOAP/REST tools and web browser tools. The output of those tests can be saved in files and databases.
Every imaginable source in the entire world of information technology can be accessed and used.
The following features are particularly valuable to us:
- Generating SOAP/REST requests
- Validating responses against WSDL
- Scripting parts of an XML request
- Accessing DBs with JDBC
- Jenkins Continuous Integration
- Regression test automation
- Modular scenario setup
How has it helped my organization?
Our tests are run nightly on every environment using Jenkins CI.
We moved from the Waterfall/V-model to Agile a couple of years back. With SOAtest, we are able to test features in every sprint, quickly and efficiently.
We use SOAtest to test the interfaces of maybe a dozen back-end services, simulating multiple, different points of sale. For example, a customer web shop, a sales agent application, sales machines, or mobile apps.
Service interfaces are continually integrated and updated within SOAtest. We test most front-end features first with manual tests, and then integrate them with validations into SOAtest scenarios.
Before using SOAtest, we had a huge delay in acceptance testing due to the amount of manual regression tests that we had to do.
What needs improvement?
Sadly, almost everything needs improving. Don’t get me wrong, the tool is very powerful and immensely useful for day-to-day usage. But certain things are missing or are set up in a way that makes it unnecessarily complicated. Here are some examples:
- Replacing part of an XML request by script requires a variable in the scenario folder. This variable needs to be prepared before the SOAP request tool with the partial XML to be replaced in the request. This variable can then be set to replace a subtree of the XML request. It would be easier if we could directly set a script in the SOAP request tool as a replacement.
- Enabling/disabling an optional element of an XML request is only possible if a data source (e.g., Excel sheet) is connected to the test. Otherwise, the option is not available at all in the drop-down menu.
- True parallel testing is currently impossible.
- There is a separate product ‘Load Test’ that has parallelism built in. But this is only to multiply a scenario with all test cases by a number of users executing all test cases. In a future release, there should be the option of running all test cases of a single test run in parallel to speed up testing.
- To build a modular scenario, you can reference test suite files and execute the test tools contained therein. But every test contained in the referenced test suite file will be executed, as it is set up in the test flow logic of this scenario.
- Sadly, there is no way to setup single operations of a web service and then combine those operations into a scenario. This is how SoapUI does it.
- SOAtest is a plug-in for Eclipse. But actually, it comes bundled with its own Eclipse and several other Eclipse plug-ins like PyDev. Although this ensures compatibility of all contained tools, it also means that every part is a little out of date. For example, PyDev is available as v4.5.5, which means you get Python v2.5.2.
- You can manually upgrade to a newer version, but this means every installation using your scenarios, and especially scripts, must be updated as well.
For how long have I used the solution?
- We have been using v9.10 for about six months.
- We have used SOAtest itself for around five years (since v9.3).
What do I think about the stability of the solution?
We had a stability issue, but Parasoft support was always very helpful in finding a solution or a workaround.
How are customer service and technical support?
I would give the technical support a rating of 10/10, or even 20/10!
No matter the problem, from simple user errors to complicated Eclipse issues, the support team is always quick to respond and is very good at what they do.
If you actually encounter some issue in which the first or second level support really can’t fix in a couple of minutes, then the third level support will dig deep and come up with amazing ways to help you.
There is a user driven support forum where you can get help and examples from other users.
Which solution did I use previously and why did I switch?
We did not use another solution previously. We do use SOAtest and SoapUI in parallel:
- SoapUI is used as freeware by developers for single/simple SOAP requests.
- SOAtest is used by the testing team to build extremely complex scenarios.
How was the initial setup?
Installation is easy and straightforward, as is the initial scenario building. SOAtest gives you the ability to choose how complex you want to build your scenarios.
The simplest level of usage is a full graphical view of requests and responses, storing values in data bank tools for use in later SOAP/REST request and SQL tools.
The more intricate levels consist of scripting in several available languages and really revealing the beauty of SOAtest.
The documentation guides you through several examples and helps enormously with the initial steps of building your test scenarios.
Which other solutions did I evaluate?
We evaluated SmartBear SoapUI.
What other advice do I have?
- Learn XML as well as XPath.
- Learn Jython, a variation of Python running in the JRE. It gives you the option to import Java classes into your scripts and use the latest Java improvements, but without the type safe limitations and language complexity of Java.
- Implement your scenarios from the beginning with continuous integration in mind. For relative paths everywhere, abstraction in SOAtest environments etc., use a version control system to save and version your scenarios and scripts.
- Take a short course (less than one day) from the Parasoft support team to familiarize your test team members with the product very quickly.