What is our primary use case?
We are in the middle of a pilot, an evaluation. We have been evaluating the software for about 13 months.
Our focus, at the beginning of the evaluation, was to probe a BDD approach with Gherkin to help us track the end-to-end process from requirements to test automation, without leaving quality aspects behind. That's the first use case. The second use case is that we want to optimize the traceability and integration within the Continuous Integration and Delivery process.
How has it helped my organization?
Throughout the evaluation, we've sensed that we have better collaboration among these three roles, the business, the testers, and the testing automation engineers or the developers. From a functionality point of view, we have been able to execute more testing automation on this particular pilot team because of the integration between the tools.
These are things we can see from the daily work between the teams. It's a feeling that everything is moving a little bit faster, it's a little bit easier.
Right now, we have a lot of tools facilitating the BDD syntax but they are scattered. For example, the product owners are working with JIRA or other application products, like Confluence. The developers are working inside their IDE. And the testers are working somewhere else, for example, on ALM.NET.
We thought the possibility of having a single platform which can connect all three roles, centralize them into one platform, would be helpful. It's quite difficult because if you write the feature files first in JIRA, for example, you have to copy the content of the feature files to the testing tool. With ALM Octane, it is possible to have synchronization between JIRA and ALM Octane, if we are using both tools. But there's also the possibility that we use only ALM Octane for both the requirements management and the testing.
By clicking a button after we have the feature files in ALM Octane, we can send a message directly through the IDE plugin of ALM Octane - for example, for ItelliJ or Eclipse - to inform the testing automation engineer that this test is ready to automate.
In other words, it would allow us to keep each of the roles flexible. So the product owner could work in JIRA, the testers in ALM Octane, and the testing automation engineers in their IDE. But we wouldn't leave the consultation aspect behind.
What is most valuable?
It's hard to say which features are the most valuable because all the features are really fantastic. What distinguishes the ALM Octane from ALM.NET is, first, the Pipeline module because with the Pipeline module you are able to integrate with the CI server.
The second point is the support for the BDD Gherkin syntax. It's valuable because the point of BDD, I'm talking about the "culture," is to optimize the collaboration between the business - the product owners - the testing, and the development. We are pushing them to speak the same "language."
What needs improvement?
First, the Requirements Module could be better, to build up a better requirements process. There's a huge improvement from ALM.NET to Octane, but it's still not really facilitating all the needs of the product owners, to set up their requirements in Octane.
Second, because JIRA is a leading tool for both development and requirements management - everybody is using JIRA - I'm pretty there will be a use case where people are trying to connect between ALM Octane and JIRA. The back-end configuration of the synchronization with JIRA could be simplified. The architecture is really complicated. We required a lot of machines to build the cluster and the configuration was not really clearly described within the documentation. This may have something to do with the fact that the software is pretty new.
I addressed this with the vendor, but a solution was not really provided. However, I saw just today that they are creating a collaboration platform for people who are evaluating ALM Octane. That's a good start to facilitate this but, as I said, because the software is pretty new - it's only two or three years old - I expect that some things are not really completely optimized. I'm pretty sure it's going to be better in the future.
For how long have I used the solution?
What do I think about the stability of the solution?
In terms of performance, I haven't had any complaints. It's really performing well. But I'm not really qualified to judge it because, for the last 13 months, we have only been working with a handful of people, with one team, some 20 users at most. For 20 users it has been really stable. We'll have to see after we have, say, 2,000 users working all at once in ALM Octane, how the software actually performs.
How are customer service and technical support?
For this particular software, the support is very good. We have direct access with an R&D colleague from the provider. They are not only reacting to our requests but also providing some insight and tips about the tool.
Which solution did I use previously and why did I switch?
I wasn't really looking proactively for a new solution at that time, two or three years ago. We were aware of the limitations that ALM.NET has, that's it's too rigid, too complicated, and the user interface is not too user-friendly. There was an announcement from Micro Focus, that they were going to release a new tool that would be the next generation of ALM. That was the first time that I heard about this software.
How was the initial setup?
It was pretty straightforward. Everything was written in the documentation, down to the smallest details. The package was as an RPM package which was good for our administrator in doing the installation. The only thing that bothered me was the configuration of the .YML file. It was actually really simple, according to what they described about what to configure there, but there were some delicate points that we had to pay attention to. Other than that, everything was really good.
The installation itself only took our administrator a few minutes. If I hadn't had problems with the .YML configuration, it probably would have taken me a couple of hours to complete the installation.
The onboarding, the transition from the old tool to the new, is quite a challenge though. We have been using ALM.NET for ten years or more. We are still finalizing the pilot, but our thought is that if we go to Octane, we would prefer to go with a greenfield approach. It's not that we're going to migrate stuff from ALM.NET to Octane. We will just start fresh, from scratch, in Octane.
The reason is that the tool provides really good functionalities for us, especially for testing. It's good to take a chance. There will be a review process and we'll try to really integrate the process with the tool.
For us, the initial setup involved three to five people, until the application was ready to be used. We have been maintaining it for 13 months with two people, myself and the consultant.
What about the implementation team?
For the deployment, for installation and configuration, we did everything by ourselves.
But we are working with a third-party, a consultant, for the customization. Compared with ALM.NET, ALM Octane doesn't have a full scripting capability, so everything is really defined as business rules. This is quite a change for us. Therefore, we need a partner to provide us with some guidance and tips.
The consulting firm we're working with is profi.com AG. Our experience with them has been really good. I have no complaints.
What was our ROI?
You have to take into account all the costs. Right now, we are using ALM.NET. If we decide to go to ALM Octane we will have migration costs, we will have costs for integrating other tools. If we stay with ALM.NET or go for open-source tools, we have to evaluate the same cost factors.
My guess is, if we go to ALM Octane, and considering all the features that ALM Octane provides, with the open architecture, that would mean we don't have to buy more plugins. We could gain some financial advantage from implementing ALM Octane.
What's my experience with pricing, setup cost, and licensing?
It will be as expensive as ALM.NET, if not more expensive. But here's a good tip: If you have ALM.NET, you are able to share your licenses from ALM.NET to Octane. You just have to define a dedicated number of licenses on ALM.NET and then you can share them with ALM Octane, with some configuration effort. This is something that you have to take into account, that there is a possibility of such license sharing that could decrease your costs.
Compared to open-source tools, the price the ALM Octane is definitely higher, in terms of the licensing cost.
Which other solutions did I evaluate?
We are doing a lot of other evaluations; for example, the possibility of using direct JIRA backends for testing. In addition, but not in the same magnitude as our evaluation of ALM Octane, we're still looking at the possibility of holding on to ALM.NET longer.
What other advice do I have?
As I said, it's best to involve all the stakeholders, for faster implementation, because it's new software and they need to be on the same page and have the same understanding of the software's concepts. In addition, assess what you need, your process, and if the tool can fit into that process.
What we did was create a requirements catalog, with a list of all the requirements from the non-functional point of view and the functional point of view. Then we started to evaluate the software based on these requirements, which were created together with the stakeholders. We had interviews with them. That was very helpful because, in the end, you have to see that the tool is providing value for you, based on your requirements.
I would recommend that you do a pilot with a team that is mature enough to work on the tool. Instead of just looking at webinars, it's better to have a pilot with a team that is really able to work on the tool. That way, they can really see, first hand, how the tool is working, if it's going to be able to be integrated with the process.
We are trying to implement Agile methodologies in DevOps right now. In terms of how our tools and processes are evolving to adapt to the change from traditional Waterfall development, it's quite difficult because we have been working with the classic Waterfall method forever. It's not just about the tools, it's about the process first, and that the people have to be on board with it. In my role, what I can provide is delivering one tool that is able to support this transformation.
We are evaluating the possibility of Octane replacing ALM.NET because ALM.NET does not really support Agile software development and continuous testing and because the workflow process, itself, is too rigid. In addition, the effort involved in the maintenance of the application is really big. That is especially true when talking about the software updates. And then, ALM.NET has a complex UI, it's not user-friendly. In addition, there's no lightweight integration possibility between ALM and open-source tools.
If we look at these features that ALM Octane provides, and that ALM.NET doesn't have, that is one thing that we can contribute, from the tooling point of view, to support the transformation. But you cannot easily say that the transformation of the whole organization depends on just one aspect, like tooling, because it also involves the process and the people.
When it comes to the biggest lessons learned about adapting tools and processes for Agile DevOps, I think it is really important, in the scope of evaluating ALM Octane for a transformation, to have all stakeholders on the same page, and to have their opinions and experience included. In addition, define the process first and then go on to the choice of tool.
Regarding how ALM Octane can help us with the transformation from Waterfall to Agile, we'll still have both methodologies. We're not cutting off one method. We'll have to live with Waterfall and Agile. But for me, Octane will be like concentrating on the core competence, meaning we eliminate waste in managing the software application, for example, by simplifying the workflow. That is important.
One issue that people forget, when comparing, is that if we are going to update the ALM.NET software, we need at least three hours to do it. With Octane, it took me one minute to update the software. That kind of waste with ALM.NET can be avoided.
The second issue is that it's important to consolidate information, especially from the testing, defects, and requirements areas. Right now, with ALM.NET, it's not possible to integrate it easily. Everything is possible. You can always do something to create integration between two tools, but it's going to take a lot of effort and resources. In the end, it's all about money. If we are able to consolidate information under one roof, by using ALM Octane and its lightweight integration feature, that will help us with the transition from Waterfall to Agile or, at least, from Waterfall to both methodologies.
After we are done with the evaluation, the next step will be to deploy it for the organization. We are piloting this software with one scrum team. The next step will be to bring more scrum teams on board with us.
I rate it an eight out of ten and, for a new product, that's quite good. Overall, it's really good software. I haven't seen anything like this in a long time. But there are some limitations right now: What I mentioned about the architecture of the JIRA synchronization, that it could be simplified; and the documentation could be better. Those are small things that could have been better from the beginning. Other than that, I really have no complaints about it. Those are just some configuration and set-up things that could be better. If those factors are eliminated, I would give it a ten.