What is our primary use case?
We are using it for test management. We have distributed teams in three locations with one location in Portland, which is the newest, and also in India. We have a team of around 150 people (developers plus three testers). We are implementing an order migration legacy system to a new system based on AngularJS 5.0. We also have test automation being implemented on this account using Micro Focus UFT.
Automation is triggered through ALM. We have the test scripts stored in ALM that are triggered through the execution dashboard. Also, the reports are available on the dashboard.
We do defect management through ALM, which is the typical use case. The defects are raised in our different locations, then the collaboration between the development leads and testers happen through ALM.
We use the Test Plan module where we have test cases related to all our different releases up until now with a few current releases as well. We use the Test Lab tab to pull test cases from Test Plan and do executions accordingly. We have also created some smoke and sanity testing suites where we pull test cases, then execute them when required during the project phases.
How has it helped my organization?
Any user who accesses a project gets to know what is the latest status on a test case, from a test case writing or test design perspective as well as test execution perspective. Collaboration is very strong. The communication that the tool sends out along with the log which is maintained is locked in the history. This is for any change at the test case level or within any of the components of ALM. The history helps us to understand what went wrong or when has somebody made a change. Therefore, the history log is a very important feature.
From a collaboration perspective, I can send out emails directly from ALM that, at times, get triggered automatically. If you raise a defect, then it automatically triggers to a particular email ID that the defect has been logged in ALM. This helps to get immediate visibility or attention of the development team from a testing team's perspective.
Initially, we used to lose a lot of time in collaboration. If we do this in a very crude way through Microsoft Excel, then there would be a lot of issues related to version control. Like somebody might say, "I've fixed the defect," and the other guy would say, "It is still open." Now, across the team, we have one single source of truth because ALM helps the whole team to understand the exact status.
What is most valuable?
Ease of use is definitely one of the strongest points for ALM. It's a very user-friendly tool and the maturity of processes within ALM are amazing compared to other tools. Their in-built reporting does help with getting ready-made reports from the tool.
The Test Plan and Test Lab setup helps us a lot when pulling test cases repeatedly from a different perspective. If I want to make a sanity pack, then I can pull test cases from that same library of test cases. I don't have to create them again or copy and paste them.
I love linking/associating the requirements to a test case. That's where I get to know my requirement coverage, which helps a lot at a practical level. So, we use the traceability and visibility features a lot. This helps us to understand if there are any requirements not linked to any test case, thus not getting tested at all. That missing link is always very visible, which helps us to create our requirement traceability matrix and maintain it in a dynamic way. Even with changing requirements, we can keep on changing or updating the tool.
We use the dashboard and have created our own reports. The typical dashboard also helps us a lot to understand test execution progress and the percentage of open defects from a defect perspective. We use the defect aging reports a lot. This saves us lot of time and gives us the right input from the perspective of which defects are aging. Those need to be looked at again and possibly discussed in further detail in the defect triage call about what's the blocker to get them fixed and how we can work in a better way to avoid the defect aging in these manners.
The vendor is still investing in the product and releasing valuable features. For example, there has been improvements in the overall folder structure. Initially, we just used to have Test Plans and Test Lab. Now, we have the Task Board.
What needs improvement?
Sometimes I do run my queries from the admin login. However, if I want to reassess all my test cases, then I am still doing this in a manual manner. I write SQL queries, then fire them off. Therefore, a library of those SQL queries would help. If we could have a typical SQL query to change the parameters within test cases, then this is one aspect I can still think that could be included in ALM. Though they would need to be analyzed and used in a very knowledgeable way.
For how long have I used the solution?
What do I think about the stability of the solution?
It's very stable. We haven't had issues with any sort of stability issues, e.g., no downtime.
What do I think about the scalability of the solution?
We have around 4,000 test cases in ALM, so I don't think scalability is an issue.
We have around 150 users. The hierarchy of ALM users is:
- The admin
- Process leads, who are using it.
- At the lowest level, there are data developers and data testers who access ALM.
70% of our people use ALM and the other 30% don't need to be on ALM.
How are customer service and technical support?
The technical support is nice. At times, we have needed to wait. However, this is understandable for a few of the issues as they sometimes can be tricky. I would rate the support function as above average.
The turnaround time varies with the issue, but they're decent enough. The average time is two days. They provide us local support.
Which solution did I use previously and why did I switch?
I have always used different versions of ALM. I did not previously use a different solution before using ALM.
How was the initial setup?
The initial setup was pretty straightforward. The process was just about our customization, which we do from our end and is admin guided. This took us around a couple of weeks and wasn't cumbersome at all. ALM is a mature product. We could set up how we wanted to upload our test cases, then structure the different parameters or columns the way we wanted them. The process was quite streamlined.
What about the implementation team?
We did have some internal help, but we didn't have a full-time consultant. We didn't have any external help. We used a team of three people (part-time consultants).
What was our ROI?
We definitely feel that it has given us a huge advantage from a collaboration and time savings perspective.
It can reduce the wastage that happens in collaboration activities. The effort has definitely gone down. Effort and collaboration have been reduced by 60 percent.
What's my experience with pricing, setup cost, and licensing?
It all comes down to how many people are going to access the tool. When teams go above 20, I think ALM is a better tool to use from a collaboration and streamlining perspective.
Which other solutions did I evaluate?
At a process level, the maturity within ALM is at the highest level. Now, if I have run the same test case five times within Test Plan, it will gives me a status of that test case based on the last run, whether it passed, failed, or its situation. If I want to know right now from a functionality perspective what functionalities are working for me and which are not, then based on the immediate last run done directly through Test Plan, I can understand that. That's one of its strengths. This is not available in other tools, like TestLink and Jira (we are using both).
Jira has an advantage from agile perspective. For an agile project, it helps to have the dashboard in the way Jira is structured.That's where Jira is pretty useful. We also have three of the defect calls running different ways using Jira. There are a few things from a visual perspective where Jira poses some advantages over in ALM.
TestLink is pretty similar to ALM. It is not really drastically different. It's open source and doesn't have the kind of maturity which ALM has, like the BI page, the history log, or other functions that are present in ALM. It doesn't have that type of strength. However, since it's open source, at times a couple of our clients use it, but I use it very rarely within our projects.
What other advice do I have?
Security is driven by the different user login credentials that are created by the admin. This is pretty typical. In this aspect, all their tools are good.
For risk-based testing, I used to have a different version of ALM that gave me a confidence level. Currently, I don't think our company has bought the version where you implement risk-based testing. However, it does help me to get the required inputs from the tool. Then, I have my own way of going about risk based testing.
I have seen the Single Sign-On. It's nice, but we don't use it in our current project due to a few constraints and a few user experience related issues. Sometimes, people don't want to change and just want to do it the old way. That is why we stopped using it.
I would rate the solution as a nine (out of 10) to keep pushing them to include more features.
Which deployment model are you using for this solution?
Which version of this solution are you currently using?