What is most valuable?
Let me walk you through our testing strategy. We have over 2000 test cases, and if I need to change a couple of things and test a cycle, it takes 10 people and maybe three to four weeks to test it. If any change needs to be made, it takes three to four weeks to see the results. That’s too much in terms of resources. Now, the automation that that we have seen from ZAPTEST has allowed us to classify cases as High, Medium and Low. We have around 10% we classify as critical so we still do critical testing manually.
How has it helped my organization?
On average, I get two cycles per week and that only requires the attention of two people. It saves us a lot of time and gets us to production quicker. In the initial stages we had three, maybe four, people who were involved with the product for around three months. At this time we had WebEx sessions with ZAPTEST consultants. This involved five days scripting training. Initially it was very slow; we were churning out slow scripting. But once we got hands-on experience, we moved very quickly. Now we are bearing the fruits of our hard labour.
What needs improvement?
In terms of improvements, right now we are using automation testing and we had an issue where we wanted to also use load testing which was not initially a part of the ZAPTEST suite. This led us to continue using HP products to be able to run load tests. After consultations with ZAPTEST, they addressed our needs and developed a plugin for load testing and now this works great. We have no issues and, best of all, they didn’t charge us for it. So our company benefited from this change.
How are customer service and technical support?
The pricing model allows us to purchase support hours, and I have purchased 500 hours. If we have any problems, this allows us to have someone come and see us on-site three times a year, allowing also for training. Technical support is available all of the time through WebEx. We send the logs to them and 98% of the time they fix the issue the same day. We are happy because they pay attention to their customers.
Which solution did I use previously and why did I switch?
When I took over the automation testing area in 2013, we were using HP products. We had two problems, however. One, we were testing Oracle products using them, and two, we are in the retail business. I wanted to test the application on registers/POC, but the HP products were not working out well for us. That’s when I contacted ZAPTEST. I knew we needed automation testing solutions, so after we got the ZAPTEST, it was a big plus for us. We now use it every week.
How was the initial setup?
The initial set-up was done through a telephone call. It was very easy. They also have videos which we used. It started with simple test cases, and once we knew what worked well, we discovered it wasn't hard at all.
What about the implementation team?
We did it in-house, but there was some additional training was required later on to do the complex test cases. I also wanted to create a chain of 700 scripts and we divided them into 10 chains which they helped with. They put everything together, how to schedule, and then showed us how to analyse the results.
What's my experience with pricing, setup cost, and licensing?
Price-wise, HP products cost us millions and with support, we paid an additional $600,000. At ZAPTEST you have a different testing model. You don’t pay a lot of money upfront and you don’t pay an increasing fee year-over-year for support. I believe they are a lot cheaper and we have seen great savings. It’s because of this that we are looking in to different modules that ZAPTEST offers, such as performance testing.
Which other solutions did I evaluate?
When I looked at other products, there were only two out there that will work on registers. The one we didn’t go with wanted us to purchase many add-ons and servers. This would have cost us at least another quarter of a million dollars. ZAPTEST worked out really well as we didn’t have to purchase any additional software. It is client-based and server-based. It worked out really well for us with the decision to use ZAPTEST.
What other advice do I have?
We are also partially a government agency and we have to follow certain security procedures. My straightforward advice is to get plenty of support and training hours included in the initial purchase for at least two years. That’s the biggest advice I can give. Any product I buy I use their evaluation license. I have three days to try it out; to make sure I like it, test simple scripts and make sure it is working. This will give you confidence and ability to compare to other products. Don’t cut corners when it comes to your training and support hours. I am trying to expand in 2016 in order to introduce a test case mapping module and once I find a good ALM, I will have a perfect match.