What is our primary use case?
We are using Qualibrate for our testing strategy, as well as for standard change requests.
We began working with Qualibrate for testing our on-premises SAP solution. After that, we started implementing the cloud application, SAP SuccessFactors, so we started using it to test that.
Regression testing is very important for this application, and we use Qualibrate for this, as well as to perform user acceptance testing.
Another activity that we use Qualibrate for is the automation of standard changes. These types of changes are pre-approved, and via Qualibrate, the configuration is done. The flow receives requests for changing an item in our ticketing system, ServiceNow, via our middleware system, which is connected to Qualibrate. In Qualibrate, existing recorded scripts will be started and then Transport in the SAP development system will be created automatically.
In the SAP landscape itself, the Transport will be moved automatically with all checks through the landscape. Of course, at the end when the testing is completed in the production environment, there will be a message sent back to the ticketing system via the middleware and the ticket will be closed.
How has it helped my organization?
I recently created a business case for Qualibrate and it has to do with regression testing. We have approximately 300 to 400 SAP end-users on-premises in the organization, across different business units. If you have to organize a regression test, it is because you have changed something and you want to know if it will have an impact on the processes in the other regions. In cases like this, you want to check to ensure that nothing was touched.
Normally you invite maybe 50 key users to test in the different functional domains. This will perhaps cost an average of four hours per person. Instead, you are running automated test scripts that have been pre-recorded. Even if the length of time per test remains at four hours, you are saving that much in terms of what it would cost to pay testers for the same work.
It is not possible to test everything, especially if you have interfaces with other external systems, but you do win a lot of the time. Also, it's not only for testing when your project is going live, or when you're bringing changes to an environment but also checking your daily operational support. It brings a really big benefit in terms of keeping your system stabilized. In the end, you have fewer disruptions for the business.
The low-code approach enables users to onboard quickly and then use it in their daily test creation, but I would always suggest centralizing it. The users have the scripts and you only need one or two people on a central level to keep control of it.
People can make use of Qualibrate for training because that's also a benefit. If you have a new employee, you can show them in the system how one of the end-to-end processes looks. You can also explain the UFT in the same way.
What is most valuable?
The most valuable feature is that it's user-friendly. It is really easy to use and if you have your adapter downloaded, you can remotely explain to your key users, the testers, how to use it.
We have all of the biggest parts centralized, which means that we can run all of the scripts at the moment we bring changes into the UFT environment. They can easily be run in the background.
You have the ability to reduce all of the recorded transactions so that you can build your own consolidated end-to-end process. If something in the end-to-end process changes then it is very simple to insert a new part into it.
We have just finished all of the scripts for SuccessFactors, which is an HR application. Every quarter, you have a new release, and we are dealing with scrambled data that needs to be reduced. Using Qualibrate makes it easy.
What needs improvement?
What I would really like to see is if you are running scripts in Qualibrate, and there is a defect, then you can have it automatically raise a defect in your own ticketing system. We do not use the defect system in Qualibrate, which is why this would be helpful. We won't want to have to use the ticketing system in Qualibrate in addition to the one we use now.
The reporting can be improved. For example, the reports can be better and show details like how many of your scripts have been executed, what the history is, which scripts have not been executed for more than a month, and other types of statistics.
It would be nice if when you add or change some specific script in your SAP system, it would automatically check to see what it implied with respect to Qualibrate. For example, it would link to the appropriate Qualibrate script if it needed to be changed. In the context of an end-to-end process, which has already been recorded, if next week I change something in the process then I would like to get an alert that we need to make the relevant change in the process.
For how long have I used the solution?
My company has been working with Qualibrate for two years.
I worked with Qualibrate for another company approximately six years ago.
What do I think about the stability of the solution?
Stability-wise, so far so good. Last week, there were two times that the system was unavailable but that really does not happen very often. In case it does happen, the communication from their support desk is very fast. They inform you directly.
What do I think about the scalability of the solution?
Scalability-wise, this solution is fine.
We don't have very many people who are working on it. There are two external, my test manager, and myself, so four in total.
How are customer service and technical support?
We have no issues with the technical support. When I have a problem and the system is not available, I want to know how soon I can get it back up and running. I do not run these kinds of systems every hour; even when your SAP is running 24/7, your testing does not. However, when you want to test something, it needs to be available, and support takes good care of this. When I have called they pick up the problem and they resolve it.
Which solution did I use previously and why did I switch?
Approximately six years ago, I was with another company and I used several different test systems. I was involved in a pilot at the time and helped to define the requirements. One of the important ones was to keep it fair user-friendly, which Qualibrate is.
Over the period of a few years, I looked at test automation. At one point, in 2018, I compared the features in Qualibrate with what it had four years prior, and there had been new guidelines for things like data introduced during that time. A lot of improvements had been made to the product.
I have experience with HP QTP, but it is an expensive tool and I explained that I didn't need a Rolls Royce for testing. Rather, I just wanted a simple tool. I didn't want to have a very expensive tool where I only use 50% of the functionality when instead I can have one that's user-friendly.
Comparing Qualibrate with my other tool in terms of manual testing and the process of documentation, I would rate Qualibrate a nine and the other tool a five. The other tool was very technical and not user-friendly. Every time you need to change something, you have to contact technical support. It is also very complex when you want to explain it to your key users.
What about the implementation team?
I have my own test manager who is responsible for the SAP on-premises solution, which includes testing it with Qualibrate. We had an external team help us with supporting the functionality of our cloud system, and we also asked them to take care of Qualibrate. Unfortunately, it took a really long time and the person responsible for it has now been replaced. Together, the new person, working with my test manager, has it under control.
If you have it under control, you just have to do the maintenance.
What was our ROI?
We have seen a return on our investment. There are a lot fewer people involved in our testing and the system is more stable. In addition, more people compliment you and less complain if the system is not available, which is a really big benefit.
What's my experience with pricing, setup cost, and licensing?
We signed a three-year contract and the pricing is in line with our expectations.
If you look at the cost of your license and what is required, you will see that it is better to keep everything centralized and pay for perhaps five user licenses. This is more cost-effective than giving one to everybody in the business.
Which other solutions did I evaluate?
When I proposed to use Qualibrate, they were hesitant because it was not in the top four or five of Gartner. That necessitated some more evaluation which included a tool by IBM and at least one other one.
What other advice do I have?
My advice is that if you know how an application is working then reduce it on a weekly or perhaps daily basis. If you are doing this then you will very quickly understand how the system is working.
The first time you record a regression test, I would advise doing it in a classroom session. In this scenario, people are doing the regression test manually and you ask them to execute the same script, but this time, start the Qualibrate recording function. People can make use of the backspace functionality and with it being recorded, it can be used the next time a regression test is needed.
I suggest that you not only look at the price but also the quality. This vendor listens to you and if you have requirements or need help, they are always available and not only by phone. If required, they will visit the customer from time to time. The application is user-friendly and there is documentation available. Overall, we are really happy with it.
I would rate this solution a nine out of ten.
Which deployment model are you using for this solution?