What is our primary use case?
We're using it for test management, to write test cases and we have put it into an overall approach which is called BDD, Behavior-Driven Development. Within BDD we're using Octane to manage all tests, to plan and do test automation. We're doing test automation with IntelliJ, together with TestCafe, which is a pretty nice test-automation tool. We have Jenkins with a pipeline connected to Octane, working the whole process.
The main intent is to have a quality solution. Our development is working in JIRA, which means that we have split it. We import user stories from JIRA to Octane and start working from there on our testing.
How has it helped my organization?
We started from scratch because we didn't have any BDD approach. We used a more old-fashioned method of development, more Waterfall and so on. We were looking for a solution which would be a good tool for our new methodology. For us, this was a key benefit from Octane, to get rid of the old style. We are implementing Agile methodologies in DevOps, that's the main thing. We try to use shift left to test much earlier and therefore it's really helpful to have Octane and to implement the new approach with it.
What is most valuable?
The most valuable feature of Octane, in comparison to ALM, is the whole Agile approach, that we can run it with the sprints and have it better connected to the whole development process.
The other feature is the pipeline, so that we can now connect to Jenkins and then have all the results from testing, from external, in the tool, so that we can see the whole approach from there.
Many things are nice so it's hard to say what is best; for example, the way it's organized. We can work with labels so we have better filtering solutions than in ALM. And it's much smarter and leaner to use than ALM.
What needs improvement?
There is a lot of room to implement new things. I would like to have more possibilities for doing test automation directly in Octane or to see UFT scripts within Octane. The implementation within ALM of UFT was much better than in Octane because I can only see the results from our test runs in Octane, but not the test itself. With Gherkins in there, that's fine, but I do not really have a hand on the scripts themselves. I can follow the pipeline jobs in Jenkins, but I can't see what's really happening there. So I would like to have some more information about that in Octane.
Also, while there is a Requirements Module in Octane, it is very plain. It's okay to have some requirements described there, but it's not really following the whole BDD approach. I would like to have more features for requirements in there; not as much as in ALM because in ALM it was a mess, it was too much.
There is a whole lot of room to improve.
What do I think about the stability of the solution?
It's very stable. Over the last one-and-a-half years, we have really not had an issue. It's fine. We only have a handful of people who are working with it so far, but for them and for us it's okay.
We have a very small team. Some are working from Croatia and we have a team here. We have a maximum of three or four concurrently using it. In the future, we will have many more. For the moment it's a pilot, but as a pilot, it's working fine.
How are customer service and technical support?
Tech support is very good. We are in direct contact with the guys from Tel Aviv and it's very good. We're in touch by phone, email, and we have sessions together with them. It's much better than it was with HPE, in my opinion.
It could be, because we are a pilot project and one of the first here in Germany, that we have more direct contact. But this is working. I really like it.
Which solution did I use previously and why did I switch?
We figured out that with our BDD approach, and what we planned with more modern technologies, and our shift-left approach, that ALM didn't fit. There was no chance to implement any Agile approach in ALM. We started to think about what else could be useful. It was pretty obvious that Octane was the right tool.
In ALM you do not have any flexibility to model the processes, to say how you would like to see new things like quality stories. With Octane you can add pipelines, and with the API you can add other tools and, therefore, other processes. All of that is not available in ALM. In ALM, you have a closed system and you have to live with what is given, while in Octane, you have the chance to add new stuff to the tool, like reports from outside, etc.
How was the initial setup?
It was really straightforward. It was a pleasure to implement our approach with it, compared to ALM or to the older tools. It was really easy. I started from zero with Octane. I had never seen it before. It was brand new and I really learned it on my own, everything in there, including the setup and how to implement the processes, etc.
In terms of maintaining it, right now it's just one person. Once we scale it, I don't expect it will take many more people to maintain it because it's very easy to maintain. If it's set up well there shouldn't be too much work to do there. For the technical parts, we will still need only one person and, within the project - depending on the number of projects - we will need, perhaps, one guy who's taking care of it from time to time.
What was our ROI?
We have had some strong discussions in regards to JIRA, to use JIRA plug-ins and to get rid of the overall HPE/Micro Focus way, because of the money. We had some discussions about whether we could make it with open-source tools. But at the end of the day, we figured out that they're really not good, full test-management tools. The overall approach, with everything in one place in Octane, for the money, is more valuable.
Since we're not finished with the license discussions, I can't tell you the end result or numbers and figures but, in the end, it's more or less equal, from the money point of view: if you're using open-source with more consulting or if you use a tool like Octane, which costs some money, but you don't have a lot of work in terms of implementing and consulting, etc.
What's my experience with pricing, setup cost, and licensing?
I don't deal with licensing and pricing, but it's worth the money, from my point of view, because it's very good. But that's all I can say about that.
Which other solutions did I evaluate?
We looked at a few. One we're still having look at is Xray but doesn't fit with our BDD approach. We also looked at ServiceNow. That's mainly for ticketing systems but they also showed us something about test management. It was very close to ALM, to the old way, so it was also not very useful for our process. Our support is working with ServiceNow but it doesn't fit with our project.
What other advice do I have?
Think about your processes and the methods you're using for development and quality management and see if the tool fits. If yes, it could be a good idea to use Octane. I have presented Octane many times within our company and outside of the company, and I have had very good feedback and many questions about whether it is useful or not. "Can you really say it's the perfect tool?" Mostly I have said to them it's really good. If you work in Agile and if you work in BDD and Gherkin, I think it's the best tool on the market.
I have a pretty long history in testing. I started in 1999 and, since then, I have worked with all these products from HPE or, now, Micro Focus. I know all the history and the older tools and I'm really pretty happy that we have a tool now which is working in a more modern way, in a good, Agile way. It's pretty nice.
With respect to how our tools and processes are evolving to adapt to the change from traditional Waterfall development, for requirements we do not have a good tool to work with, but we have Octane for testing, we have JIRA for development, and we still have ALM for defect tracking and for working together with the other teams that are still working in the Waterfall process. So for synchronizing of defects, we are connected to ALM. We have IntelliJ for development, and we use it together with Cucumber and TestCafe for test automation. We have Git for all our results and for version control. We have Jenkins, as mentioned before and, for reporting, we are mainly using Octane. This is the overall tool landscape we have.
The biggest lesson about adapting to Agile for DevOps is that it is really important to have APIs, to have open interfaces to connect all of these tools together; to have the chance to implement the pipeline easily. We are no longer bound to only HPE or Micro Focus tools. We can work together with open-source tools. It was easy to implement such things in Octane. This was a great lesson.
For our releases, we still have a Waterfall approach. We have a live release every three months. It was a little bit tricky to put together the testing for Agile and for Waterfall so that we could do the quality assurance for both approaches in one tool. I've found a way that I can have sprints over a longer time for the UAT, using Octane. We have 40-day sprints and testing in one tool. It was really nice to have found a way to have them in one tool. This was also a good lesson, to see that both can work in one tool.
There's room for more features but, for a relatively new tool, it's very good. I would rate it at seven out of ten. If the features and enhancements we have requested come through, it will be a ten in the future. Given the maturity of the tool, that it's only one-and-a-half or two years old, seven is a very good number. I can give it a 10 when the Requirements Module is working better and when some other things are solved, some problems with implementation that need work. Then it will be a ten.