What is our primary use case?
The primary use case is to the overall testing process and management of our test cases, as far as the design, creation, review, and archiving of them goes. We use it to manage their overall status.
We are cloud users, so we've got the latest and greatest version. They transparently push updates to us.
How has it helped my organization?
The solution’s reporting enables test team members to research errors from the run results. We do have some metrics and some dashboards that are set up that which allow the testers themselves to get good visibility into where things are at and which allow others to see "pass," "failed," "blocked."
qTest has been very useful for us. It's helped in productivity. It's helped in automating a lot due to the seamless integration with JIRA. It has taken us to the next level, in a very positive way, in the management of our overall test cases. It has been outstanding.
In comparison to managing test cases in spreadsheets or other tools we've used in the past qTest is saving us a couple of hours a day.
Investing in Insights to have one location for a dashboard of all reports and metrics, it has allowed us to minimize the number of reports or URLs which other stakeholders have had to go to in order to get status on the testing. There has definitely been an improvement there.
Use of the solution also provides our team with clear demarcations for which steps live in JIRA and which steps live in qTest. Test cases and tickets are assigned to test plans, etc. through the tools within qTest and they are all linked back.
What is most valuable?
The main thing that really stuck out when we started using this tool is the linkability of qTest to JIRA, and the traceability of tying JIRA requirement and defects directly with qTest. So when you're executing test cases, if you go to fail it, it automatically links and opens up a JIRA window. You're able to actually write up a ticket and it automatically ties it to the test case itself.
It has seamless integration with other key, fact-tracking or ticket-tracking tools, with overall good API integrations.
What needs improvement?
The Insights reporting engine has a good test-metrics tracking dashboard. The overall intent is good, compared to other test tracking or test management tools. But the execution is a little bit limited. The overall solution is good, but the results are not consistent. The basic premise and functionality work fine. When you try to extend the use of it a little bit more, it struggles. It is a little clunky with some of the advanced metrics. Some of the colorings are a little unique. They are currently working on a new flavor for Insights.
We do have dashboards and links set up that our executive level access. Overall, the numbers are accurate, based on what we're putting into it, but where we lose integrity or where we lose the overall perception of things, is when the colors start changing or when red is used to mean good. That's when executives lose respect for it. We've used it as a dashboard during key deployments. And then, as press is being made and the reports are being updated, colors start to change and that distracts from the overall intent of the reporting progress.
We chose to leverage Insights so that we didn't have to manually create charts via either a Google Sheet or Excel since we don't have the resources, time, or bandwidth to do that. That is what excited about Insights. But then, it just didn't meet our expectations.
We have voiced our concerns to Tricentis and they definitely have empathy. We talk about it and they keep us updated. With an acquisition they're going to leverage their analytics tool. We are excited about that, once it launches.
We have also discussed with our account manager a couple of possible enhancements here and there, but nothing that's critical or major. One example is when you're trying to link test cases to requirements, a lot of time there is duplication between the two. Sometimes you want to tie in some of the same test cases to the same requirements. An enhancement would be a quick way to copy that over directly without having to manually link every single one again. We have some instances where a large chunk of test cases are tied, re-used, and similar. When you get upwards of 15 or 20, to limit some of the tediousness of doing them all manually, if you could take a copy of the links from one and switch them over to another, that would be helpful. It's not of major concern. It would just be nice as a quick way to do it.
Another example is that with the charts — and again, great intention — you can put in a date range and apply it. Then you get to another screen and come back. After updating several charts, the date range is gone again. You have to go back in and it's sometimes two to three times before that date range is saved.
For how long have I used the solution?
It's just about a year since we procured licenses. We've been using it for about 11 months.
What do I think about the stability of the solution?
Stability with qTest is not an issue at all. We've had no downtime and no complaints, along those lines, with anything at all. qTest, by all means, is definitely one of the top test management tools out there.
What do I think about the scalability of the solution?
We're not a big shop so for our situation it's fine. We haven't seen any bandwidth issues with running in the cloud. People are accessing this tool across the globe and we've had no complaints or issues.
We don't plan on rolling it out further until we see the analytics portion of it. Our plan is that we will pick back up again at the start of the calendar year, once we see, at the end of this year, what analytics has to offer and once we get that working. Then we'll go back to the drawing board on how we can use it and then we'll roll it out and provide training.
How are customer service and technical support?
They have been doing okay in terms of the suggestions we make. It depends on the level of severity of what had occurred, what changes are needed. But they're responsive. We do get communications from the support team pretty well and our account manager is pretty good on following up on things.
For the most part, first-tier support has to ask some basic questions, but they're pretty good. There is room for improvement on communication response time from first-tier support. What we do is we wind up copying our account manager on tech support requests so she can assist in following up a little bit quicker. Ideally, we shouldn't have to do that, but we have learned to do that and it does make it a lot faster.
Which solution did I use previously and why did I switch?
We worked with a customized plugin within JIRA, not even a basic, off-the-shelf version. It was an in-house created module that was built to integrate. They couldn't afford to buy a plug-in, so they made one. That was why we started looking for a new solution. It was horrible. I would have preferred Excel.
How was the initial setup?
Because we have used tools like this in the past, we knew what we were getting into and we hit the ground running. So the initial setup was pretty straightforward. Compared to vendors we've worked with in the past, they've been extremely responsive, especially on the client success side of things. We've had that type of support and they have made sure that our needs are met. They have set us up with training and the like and that has been a really good experience.
Our deployment of the solution took a couple of months. Our complexity was that the test cases were being managed as tickets within JIRA and not necessarily using a test management plugin. The conversion of the test cases, and ensuring they were being transferred and translated into a single entity of the test case, was quite a big project.
What we were using before was a JIRA plugin. Given the way it was designed, what we had to do was extract everything into Excel and then import things in. That part of the tool works phenomenally. It's just that we had well over 20,000 test cases to deal with. We wanted to make sure we organized them into libraries. So it took a bit of time to get everything instated in proper order; to make sure that we didn't just dump everything in there.
We had one person doing the initial deployment. On Tricentis' side, there were two people involved in training us as well as our client support person. At this point, there are just two of us who are managing the tool. We tag team, but being that I am the senior director of the organization, I've tried to become the subject matter expert. I didn't really have anybody to delegate it to. That's why it's been a challenge that Insights is not behaving for us.
We've got 50-some licenses, but we probably see a peak of concurrent at no more than between 15 and 20. We're a medium-size company with about 1,300 employees. Mostly it's quality engineers who are using it. Developers have access to help with test cases. We're trying to get scrum masters in there to use Insights but with the challenges we've had with it we've backed off the roll-out of that.
qTest, is being used quite extensively. But there are just two of us who mostly use Insights. It's good in its ability to correlate all of the results coming from a double-digit number of scrum teams from across the globe. We can see the status of that testing.
For our team, the adoption of the solution has been fantastic. It has been well-received. You couldn't ask for a more straightforward, user-friendly, easy-to-use tool on the qTest side, from a user perspective.
What was our ROI?
We have absolutely seen ROI. We didn't have good visibility and transparency.
Don't get me wrong about Insights. For basic "not run," "pass/fail"-type metrics it is fine. It gives us much more visibility than we had in the past in terms of the ability to collaborate on the design, review, tracking, and archiving of the test cases, and the basic results of some of the sprints.
What's my experience with pricing, setup cost, and licensing?
We're paying a little over $1,000 for a concurrent license. One of the solutions we looked at was about half of that but that one is very much a bare-bones test management tool.
There are no additional costs. We pay a flat yearly rate for each license.
Which other solutions did I evaluate?
We looked into SmartBear and Zephyr, and not that we would purchase Quality Center, but it was used as a benchmark.
The main reason for going with qTest was not only that their test management application is more feature-rich and a good solution compared to others, but the ability to create a dashboard and report on a ton of metrics. We could have saved a lot of money, but I pushed hard for paying a premium to get the Insights dashboard.
What other advice do I have?
The biggest lesson I've learned from using the solution, because of the Insights challenge, is that I would probably do more of a formal trial. They are aware there are issues with it, and they are going to work on it.
Absolutely use it for its test management capabilities, without a doubt, but have an alternative solution for your reporting metrics.
Your testing using the tool is not going to change the result of the testing. It's just that the means are more efficient. Our testing scope has been the same and our processes have all been the same. But we're implementing a tool that's a little more organized. We're not really going to become better testers just because we're tracking things a little bit differently. It gives us more efficiencies and an overall improvement in the transparency and visibility of testing progress and its status. qTest has been pretty rock-solid.
Which deployment model are you using for this solution?
Which version of this solution are you currently using?