What is our primary use case?
We have multiple teams working at Reflexis and test management is a critical aspect. We wanted to be able to maintain tests. We have multiple releases to be sent to customers and to internal teams.
We use JIRA for defect management and for our internal project-tracking purposes, but for test management we primarily use qTest.
How has it helped my organization?
qTest has created a lot of transparency in the distinct classes. Everyone now has access to the tool, so there is visibility, internally, from one team to another team, regarding the results. When the builds are being sent out, people know how stable a build is and what the quality of that release is like. This information is very transparent and available to everyone who has access.
The way it's optimizing things is through the transparency within the teams. For example, we have an engineering QA team and then we need to send the build release to the implementation QA team. They are also able to review things. They get to know what things have passed or failed. And when we need to share with customers or others, they get very good information. They know that these builds have taken care of these things.
With respect to accountability, it provides clear information: This person has worked on this and that defect or these and those test cases and whether they have passed or failed.
As a QA team, there is more accountability. Now we are able to see what the test cases are that are assigned to us for QA, how much has been executed, and what has passed and what has failed. Later, those things can be evaluated, so it improves the accountability of the tester and creates more transparency in the results.
qTest has improved our time to release. With the automated testing which we are able to integrate with qTest, people are able to go through things immediately. We haven't seen a big change in time to release, but there is a gradual change. It has definitely improved release time, but that still needs to improve a lot. Release times have improved by 20 to 25 percent, roughly. We expect that to increase a lot. A few teams have adopted qTest completely, while other teams have started to adopt it in their work. Those things are going on in parallel. As more teams come into qTest, release time should definitely improve, in the longer run.
In addition, the automation integration that we do has been valuable. Because it has APIs, whenever we run an automation test it is automatically updated in qTest. Those efforts have been taken care of, especially with the transparency that it provides when we need to share the results or the release status with other teams. That is certainly a big plus we get from qTest.
What is most valuable?
The integration with Selenium and other tools is one of the valuable features. Importing of test cases is also good.
The way we structure the test cases and the way we structure the execution cycles and the way we are able to integrate the requirements with the test cases and then generate reports, they're all pretty awesome.
There is a qTest reporting engine and they have Insights which is separate from the standard, conventional reports. Insights is pretty good. Once you get into it and start to understand how it has been designed, you will be able to take advantage of all the features.
The reporting is awesome. The way you get the stats and other critical information from the test reports in qTest is good.
What needs improvement?
We feel the integration between JIRA and qTest could be done even better. It's not as user-friendly as qTest's other features. The JIRA integration with qTest needs to mature a lot. We have some concerns and we have some challenges as we try to work with those features. This is an area where, if we see more improvements, we will be very happy.
We need smarter execution with JIRA in the case of failures, so that the way we pull out the issues again for the next round is easy. Currently, we have some challenges and complexities around that. Locating JIRA defects corresponding to a trait from the test results is something of a challenge. It impacts productivity. The reason is that the team spends more time on mapping it again for new execution failures. If that is taken care of, it will actually save a lot of QA effort.
I'm not sure if someone is working on that. We had raised this point during our evaluation, so it was probably discussed at some point in time, that they will get at it, but we don't have a clear version by which it will be taken up.
Also, Insights is not that easy to use for someone who has just started working with qTest. You need to know what all the fields are and have some background on Insights. It's not that user-friendly for someone who's just starting to work with it. People should be trained so they know what all the various features are inside it. Then people will be able to appreciate it.
For how long have I used the solution?
We've been using it less than a year.
What do I think about the stability of the solution?
The stability has been good. It's definitely serving our purposes and that's one of the reasons went for qTest.
What do I think about the scalability of the solution?
We see it helping us in the long-run as well. qTest seems to be adding more and more new features.
We have about 40 to 50 team members using it right now. We plan to slowly increase the number of users. It's a gradual process. We are planning to scale it. We are not currently reaching the peak of 25 concurrent users, most of the time. It rarely gets to the max. We average 15 to 20 users at any point in time.
There is no immediate plan to increase our licenses. As more teams and more members come into play, and when we hit the peak very frequently, we may increase the number of licenses.
How are customer service and technical support?
We have the option to contact tech support but, so far, except for a couple of times, we haven't had a reason to contact them. Tech support is good. They have set up a good infrastructure and process, so things are getting addressed quickly.
Which solution did I use previously and why did I switch?
Previously we had TestLink but we found many challenges with it when we had to run automated tests. There are good features in qTest, which helps us in maintaining it and sharing with others, with ease. The UIs are good and give a lot of flexibility to the testers when working with them. Those are some of the main reasons that we chose qTest for our test management.
We did an extensive evaluation of qTest. We had multiple people from Tricentis helping us during our evaluation process. It has been adding value to our organization.
How was the initial setup?
The initial setup was straightforward. I was not involved in that process. It was done by the IT team in discussion with qTest counterparts. But overall, I didn't see any challenges. It was planned for a specific day, and it was completed on that day.
There was one person from our side and one person from Tricentis involved.
The adoption has been good. The organization is impressed with the features and the value that it will add to our QA processes. That's definitely a positive. It's definitely doing what we were expecting. We haven't seen any concerns from the end-users or management.
What was our ROI?
We have definitely seen ROI. One area of return is due to the simplicity of use. It brings a defined process to the team. TestLink, which we used previously, was not very usable for the testers in terms of maintaining the test cases or creating them. It was taking a lot of time. People are able to work with qTest and are able to focus more on the actual testing, rather than maintaining things due to complexities. Those are the areas it has improved.
We haven't seen dollar savings, but it is definitely adding value to the teams.
What's my experience with pricing, setup cost, and licensing?
It is pretty costly, from what I remember. It's quite a few times more costly than other tools on the market. We compared it to the other leading test management tools. We went for it because of the features and the value it could add to our organization.
Which other solutions did I evaluate?
We have evaluated several other tools. But the features, especially the requirements being integrated with the test cases, are pretty awesome. Many tools do not have the features and, even if they have those features, they are not as simplified as they are in qTest. That's one of the primary reasons qTest has been very useful for us.
Open-source solutions don't have as many features and their usability is also not as good.
Multiple people in our company evaluated other solutions and, based on all their input, we finally chose qTest.
What other advice do I have?
Do a cost-benefit analysis. qTest is more costly than other tools. If you have multiple teams, it's going to be essential, and it's worth buying qTest. Apart from that, if cost is not a factor, there are more benefits from qTest and it's definitely a tool you can go for.
All the features we have used are pretty impressive and good. The JIRA integration is the only thing that, if it is very critical, you need to plan accordingly.
It's a good investment for the implementation of the QA process. It creates more accountability in the team and also makes a lot of things easy for the managers as well. It simplifies a lot of QA processes. These are the things we've learned from using the solution. As we start having other teams use the tool, they should also be able to see and take advantage of these things.
Not many business users are using qTest. We share reports with them and they use them for management and other purposes. Primarily, qTest is used by the QA team only. But people take the reports as a starting point for discussion for things like product-improvement purposes. The business users rarely go into the tool to get to the various details they need. Mostly the reports are PDFs that we generate. That becomes the source for them instead of them logging into it and getting information.
The IT team maintains it along with all the software that we have installed on our premises. That team is taking care of it. But we hardly have any maintenance requests for qTest. There have been a couple of times where we had outages but, apart from that, we have hardly had any maintenance requests for qTest.
We haven't seen any change in the number of defects. It mainly creates transparency, and accountability has been increased.
It's easily understandable, including the reports. It's pretty comprehensive and provides all the essential details that we need to publish from any of the teams.
I would rate qTest at nine out of ten. It's a perfectly good tool. It definitely serves its purpose and I can definitely recommend it.
Which deployment model are you using for this solution?