Tricentis qTest Review

Provides a central point of reference for tracking bugs and failures, who owns the issue and its status


What is our primary use case?

I use it for test case management. I manage testers and I use qTest in order to schedule and track test case execution within my testing group.

We're on the cloud version.

How has it helped my organization?

The solution’s reporting enables test team members to research errors from the run results. That has definitely sped up productivity because it allows multiple engineers to be aware of the failures, all at once and in one place. There's no duplication of effort because everybody knows what's going on and who's working on it, through qTest, as opposed to people seeing an email that something's wrong. In the latter scenario they might all run off to try to fix it and then you're duplicating effort through a lot of people working on it and not communicating with each other. Having qTest as the central point when there's a failure means we can easily track if a bug has been created on the issue, who owns it, who created it, and what its status is. All of those are linked right in qTest so you can automatically see if this failure is being tracked and who is tracking it.

Previously we were using a product called Zephyr. It did not have history based on the test cases. At least it didn't have the history the way I wanted to track it. It didn't show all the defects that were generated by that test case and it didn't track and display those defects statuses within JIRA. QTest, specifically with the link from qTest to JIRA — so that my test cases are continuously linked either back to the requirement that generated them or to any defects that were created because of them — that link is what is allows me to be much more efficient because I'm now not running between multiple systems. I'm not saying to my testers, "Hey, who's working on this? What was the problem with that? Why don't we run this?" All of that information is located right there in the solution. 

My personal efficiency has been increased because I have a single point of truth within qTest, to always be able to see what the status of my tests is. My team's efficiency has been increased, again, because of the lack of duplication of their efforts. They always know what's assigned to them and what they own and what its status is. And they don't have to manually connect test cases from one system to the next, because they're automatically linked and the information is automatically shared. There are a lot of efficiencies built into that link between qTest and my ticketing systems, as well as, of course, by using qTest in my automation systems. Those links are really what has turned things up.

qTest has probably doubled our efficiency. There has been a 100 percent improvement in the time the testers and I spend on managing our test cases.

We have also used the product for our execution of open-source test automation frameworks. In our case specifically, that would be Cypress and pytest. I wouldn't say that ability has affected productivity. I don't think it has a multiplying effect when it comes to doing automation faster. Its multiplier comes after you've created the automation. At that point, executing it and getting the results are a lot faster. We still execute test case automation the same way we always did. We put a JSON file into Jenkins and Jenkins executes the test cases. But now, instead of just executing them and being done with it, it executes them and reports the results back to qTest. It's the same process, just with an extra step. Because of that reporting, we have a central point of truth. We don't have to look at Jenkins and try to figure out what happened, because it's not a very good interface to get an overall view of the health of a system. That's what qTest is.

In addition, the solution provides our team with clear demarcations for which steps live in JIRA and which steps live in qTest. Using, say, requirements within JIRA to test cases within qTest, there is a distinct difference between those two systems. Being able to build off of the requirements that are automatically imported allows my people to generate test cases faster and in a more organized manner, because they're based on information that's being given to them by project management via the requirements. It makes it clearer where each step that lives within the process and that is an efficiency-increaser.

Finally, since we started using qTest we have seen a decrease in critical defects and releases, although not a lot. We didn't really take on qTest to reduce the number of defects. We took on qTest to be better organized and efficient in our quality assurance processes. I had no expectation that qTest was going to decrease the number of defects we had. It was definitely going to increase the efficiency and the speed at which we were able to do our testing. That does then decrease the number of defects and issues that we run into on a regular basis. Over the first year there was probably a 50 percent decrease and over the second year we've seen about ten to 20 percent. It's not significant but, again, it was never expected to be a significant decrease.

What is most valuable?

Among the most valuable features are 

  • test automation tracking
  • JIRA linking
  • defect tracking
  • reporting.

The test automation tracking is valuable because our automated testing systems are distributed and they did not necessarily have a single point where they would come together and be reported. Having all of them report back to qTest, and having one central place where all of my test executions are tracked and reported on, is incredibly valuable because it saves time. It allows me to just look at and use one place, and one reporting solution, to track all my executions and the defects that are generated from those.

The integration with JIRA allows us to have an integration between both our automation testing systems, such as Jenkins, through qTest, and into JIRA. It allows all that data to be transferred and distributed among all the different stakeholders within the organizations. That way I don't even have to do reporting. They can just look in JIRA and see what the testing results were. It's very simple for me. It makes my life a little easier so I don't have to generate so many reports.

What needs improvement?

I wouldn't say a lot of good things about Insights, but that's primarily because, with so many test cases, it is incredibly slow for us. We generally don't use it because of that. It would be nice. It has good features, but as soon as we started using qTest, Insights became unusable. I do know that they're planning on replacing it next month. It's the one bad side of the application and they're replacing it, so at least they're listening to their customers. They know when they've got a problem, so that's a good thing.

In addition, within Insights, the report creation could be more versatile and intuitive. Generally, the reporting tools could be made more streamlined and easier to access by people outside of the organization. If I have one complaint about qTest, it's its reporting. Again, that is something that's being replaced here soon, so it'll be an invalid point within a month.

It has already been fixed in the on-premises version. The hosted version has yet to have the replacement. I don't know what the replacement's going to be like. I haven't used it so I can't really judge it.

For how long have I used the solution?

I've been using qTest for over two years.

What do I think about the stability of the solution?

There are some optimizations that could be applied. There is a bit of lag when you're getting up into the hundreds of thousands and even millions of records, but that is to be expected. 

Stability-wise it has always been available. I actually can't think of a time when it wasn't available when we needed it. The stability itself has been 100 percent. The optimization is an area for improvement.

What do I think about the scalability of the solution?

The scalability has definitely been impressive. We've got a global organization with so many different teams and I don't hear any complaints from any of them. They're all up and running on this product, all around the world. So we've scaled extensively. The different teams don't really affect each other, but we're all using the same system. We don't really notice that there are 30 different product teams using the system. You only see your own.

It's extensively used in the sense that all the QA organizations within the different product teams — we're looking at 15 to 20 different product teams, each with five to ten quality assurance engineers, and some of them with up to 30 or 50 engineers — all of them are using the product at least as their test case management system. Some of them have different implementations when it comes to their automations. Some have different implementations when it comes to their ticketing system integrations. But all of them are equally supported by the product in different project scenarios and product configurations.

It requires zero people for maintenance because it's cloud.

How are customer service and technical support?

Tech support is incredibly responsive and has always come back very quickly and helped us find issues. They have gone out of their way to make sure that we are served as best as we possibly can be. I feel like I'm in really good hands with them. That definitely started from the time at which we took on and transitioned to qTest, in the way that they helped us get up to speed with information and support.

Which solution did I use previously and why did I switch?

We used JIRA and both the Zephyr and the Xray plugins. The scalability of those plug-ins was usually fine. They scaled along with JIRA, and JIRA is endlessly scalable. Reporting is where they would fall down. JIRA doesn't have the greatest reporting and most of the reporting is manual. When you're looking at reporting within qTest, most of it is already built for you. It has canned reports that already exist and which don't require a lot of effort. Mind you, that is where qTest somewhat falls down as well, on the reporting side of things, but it is still head-and-shoulders above the open-source solutions.

The decision to move to qTest was due to the way we had our implementation. We had no central, single enterprise-class test case management solution available to any of our teams. As they grew and became more extensive, they found either that the low-budget solutions they were using, or the open-source solutions that we're using, or the complete lack of solutions that they had, were simply not adequate. The decision was made at that time by upper management that we needed to find a central, enterprise-class solution for test case management. 

How was the initial setup?

The initial setup was very straightforward since Tricentis did most of the work for us. We're using a hosted cloud product so for us it was, "Here's your username and password." 

We received extensive support from, at that time, QASymphony, and Tricentis now, in getting up and running, understanding the product, and getting the information that we needed to make the best possible use of the product and to be successful. QASymphony and Tricentis have excelled at making sure that we are successful. I have a regular meeting with my success manager and she's always on call to be able to help us with issues.

Globally, for our organization, it took about six months for complete adoption. That was not Tricentis' fault. That was just how long it took us to get everybody up to speed and onboard. If it came down to how long it took Tricentis to do the deployment, it was probably a day and we were up and running and ready to go. There was not really a lot of configuration required on their side. The effort to get a large, global organization transitioned from one tool to another is not trivial. With Tricentis' help we were able to do it in what I would call an "impressive" six months.

Our implementation strategy was varied. Globally, we have many different projects and project teams and they all were using different tools. Some were simply using spreadsheets, while others were using tools like Zephyr. All of them chose to transition over to the central qTest test case management system. Each team had a very different implementation and that's definitely where Tricentis' support shined.

What was our ROI?

We have definitely seen return on our investment, simply through the efficiencies of the process. It's a tool that everybody knows how to use and it's global, so there's a good support network. And the support network from Tricentis is so extensive and useful to everybody around the world. Simply through the increased efficiencies of our test case management system, we have seen a return on the investment. That's not even taking into account the improvements in quality within our products, which is immeasurable.

What's my experience with pricing, setup cost, and licensing?

There is an upfront, yearly cost for concurrent licenses, meaning we're not limited to a specific number of users, only to a specific number of users online at a certain time. That works really well for us because we're a global organization. We'll have people online in San Diego, and those licenses then can be used later in the day by people online in Tel Aviv. It's been a really great licensing model for us.

I believe that there is a maintenance cost as well. I'm not really involved in the payment of that, so I don't really know what it would be.

Which other solutions did I evaluate?

An evaluation was opened up to search for the proper solution. qTest was the winner. 

What other advice do I have?

The biggest lesson I have learned is that the transitioning process is only difficult if you drag it out. Transitioning over to a new product needs to happen quickly. It needs to be a top-down decision and the information needs to be disseminated to everybody in a quick and efficient manner. We saw that happen easily with the qTest product and that sold me on the lesson that I learned, when it comes to implementing new, global-enterprise software.

qTest is a great solution. It should definitely be at the top of your list when you're looking at test case management solutions. It's really the service and support that comes from Tricentis that sets it apart. In addition to that, its integrations with systems that we are already using is the force multiplier that allows qTest to be even more efficient than just another tool that people have to use on a regular basis. It has become the only tool that they have to use on a regular basis.

In our company, executives or business users don't review results provided by qTest because that would be a kind of an Insights thing and we don't really use that. They do see the testing status in their tool of choice because we have the links to JIRA, so that they don't have to review the testing status within qTest. They don't log into qTest at all. They see the information that they want through our links with the ticketing system.

The solution doesn't really help us to quickly solve issues when they occur, but I don't really feel like that's its job. Its job isn't to help me solve issues. Its job is to make sure that I'm aware that there are issues that need solving, and that information is distributed to all the people who need it, right when it happens. There are some things in there that help me figure out what's going on and what do I need to do to fix a problem; it depends, of course, on the problem. But I don't feel that qTest's job is to help me solve problems. qTest's job is to make sure that I'm aware of the status of problems, that there are problems, and whether or not they're being worked on.


Disclosure: IT Central Station contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
Add a Comment
Guest
Sign Up with Email