What is our primary use case?
We are using ALM Octane for electronic component testing and validation. We have a few departments where they are developing their software and using JIRA projects and exchanging results with Octane.
About 80 percent of the users are not in software development itself, but are in software testing. The software is developed by external companies and we are just doing the integration testing. We are putting the components together from five different suppliers, and then doing the integration testing. Is the software working in real life, together, from the different control units of different vendors? It is a staged process. We check if things are working in the different parts of the system, like the engine components, drive train, navigation, and infotainment systems. If things are working in those different areas, we put everything together and test it in a complete car.
As a result, we have lots of test cases. We have automated tests and a test automation tool that is controlling complete car-wrecks and the like. So it's not only a mouse pointer on a screen. It's controlling robots opening and closing doors, for example.
Our main focus is efficient planning of tests. We cannot run all the tests we have every single week, because lots of the stuff has different variants for Europe and the U.S. and China. So we have to have very sophisticated test planning. A lot of attributes are needed for this and for all the runs, whether manual or automated. We have what we call a very large problem management process to work on the defects with the 100-plus suppliers that are delivering different control units and, therefore, software packages.
How has it helped my organization?
We use Octane's Backlog and Team Backlog capabilities a lot. For example, we use them for errors that occur in software that we are not developing ourselves, where we are just doing the integration testing. In such cases we are using user stories to order teams to test a certain number of test suites or test cases. We can use it straight away, out-of-the-box, without breaking or adding something to the tool. Using Team Backlogs means our teams can use all the features that come with Octane, for our benefit, without doing anything else. In the past, if you compare it to the old ALM solution, lots of teams had to store their tests and results in ALM.Net and use JIRA as a parallel system. They were manually copying and pasting links into both tools to control their workloads. Users were used to working with user stories as a work order, and that is now integrated in one tool, which is a huge benefit.
It has also reduced our testing costs. I don't have the numbers, but we're speeding up a lot. Just the waiting time we had with the old ALM when logging in was about one minute. If you multiply this by 5,000 users who are logging in on a single day you can calculate very large savings, very fast.
What is most valuable?
The key feature is the usability. It is fast to learn and easy to use. It's very intuitive to work with. Most of the important functions are available via a few clicks, compared to other tools where I have to open a sub-menu and then a sub-menu and another sub-menu, and then press a button.
The native support for Waterfall, Hybrid, and Agile software development at enterprise scale was one of the reasons why we changed to Octane. In the development process we're creating the requirement specifications which are then handed out to a supplier, including Bosch, Continental, Alpine, etc. They then develop control units with software and we have to link our tests against those requirements to check if everything is implemented. This is a very important task. It's required by law. For example, for autonomous driving, we have to prove that the car is not, by default, running into trees. We are proving that by test cases that are passed. While that is still Waterfall, it's not Agile, we are using the Agile methodologies more and more to control our workload. For example, we are using a user story in test management to order teams to test a certain number of test cases.
In terms of integrations to proprietary, third-party, and open-source tools out-of-the-box, it has a very powerful REST interface. We can interact with other tools. Micro Focus also offers synchronization tools, Micro Focus Connect Core, which has out-of-the-box interfaces to industry standards tools. For everything else, you can use the powerful REST interface, which is both good and bad. It's good for creating an interface but sometimes our engineers use the REST interface to do things they should not do. But that's because engineers are always doing fancy stuff.
What needs improvement?
The elements in filtering need to be improved, meaning the number of filters I can use in widgets or in the grid views in parallel. There's a limitation which bothers a lot of our users. Filtering in text, or having a complex filter is limited. In a given field, for example, I can use a filter only once. I cannot say, "Include the values 1, 2, and 3, and exclude value 17." This is not possible but we have requested it often.
And in general, widgets should be more flexible and more sophisticated, with the ability to layer two different widgets. There is also room for improvement in the amount of test cases which are available for certain filter conditions and a given widget, versus what was worked off already.
Also, when it comes to getting reports out of it, and maybe this is a little bit specific to the automotive industry, there are certain requirements by law where we have to export the test results for the final software delivery and create PDF reports which are stored for 15 or 20 years. Creating reports in PDF, or PowerPoint which then become PDF at the end of the day, is something which could be improved a lot. We're working on it with Micro Focus in every single release as new features are added.
For how long have I used the solution?
We have been using ALM Octane on a production scale since the middle of 2018. We started digging into the different tools about one and a half years prior. At that point our idea was to change from ALM to whatever other tool. We decided to go with Octane in early or mid 2017.
We are trying to use the latest and greatest version. We are now updating to 15.0.60, the so-called "Beatles" release, because of one technical issue that we solved together with Micro Focus. But in two to three months we will be on the latest and greatest "Depeche Mode" version which will be 15.1.40.
What do I think about the stability of the solution?
It's stable, out-of-the-box, with very few errors. And if we get an error message, very often it's because of the complex rules we have implemented, ourselves, in Octane. But in general, the usage of Octane is very good, the quality is very high, with very few errors and bugs, and with high reliability even on a large scale. We are close to being the largest Octane instance for Micro Focus.
What do I think about the scalability of the solution?
The scalability is pretty good. We did some load tests in advance, before making the decision, pro or con, on Octane. With Octane we are in a very good position. We expected to have some 5,000 users by the end of 2019/beginning of 2020 working on Octane. I believe it can go up to 2,000 users working in parallel. We hope to be powerful enough, with the architecture and everything else we set up, to meet users' expectations of performance.
We are involved in further step-by-step expansion of our use of Octane. For this year, we are planning to extend the native Octane usage of test automation, the DevOps module. We are introducing it and maybe we will be able to replace some home-grown and other tools and to integrate them into Octane to have the benefit of Octane. It would be helpful to have everything in one place for the monitoring and reporting possibilities. Our processes and needs are changing from time to time, and this is always reflected in the test management tool.
How are customer service and technical support?
I'm very proud of the opportunity that Micro Focus offered to our team, to be something of a design partner. We have a very strong relationship with R&D and are able to discuss user wishes and our needs directly with them. They listen very carefully and try to deliver solutions for our problems and enhancement requests. It is amazing to see how fast and how stably things come together, even if some users are complaining because their single feature request has been open for two years. But that's because we have more important ones. It's an awesome relationship.
You also have to take into consideration that Octane has become an industry standard. Lots of different companies are using it. From that point of view, you get to know how complex the work is for Micro Focus, and how valuable a good relationship is. I'm very thankful for being part of this kind of working together and improving the tool.
And it has often turned out that our requests and wishes have had a huge benefit for other customers.
The benefit of this design partnership, which is unique, is a plus for us. We are able to influence the tool and get features, especially within the timeline that we need them. That's one of the biggest advantages.
Which solution did I use previously and why did I switch?
We knew we needed to go with Octane for a couple of reasons. From the business side, there were some requests which ALM could not cover, in terms of data storage, etc. Since we have so many test cases, and given how versioning and base-lining work in ALM, it would require so much storage that we were not using the versioning in ALM. This was one of the biggest pain points from an ISO perspective in terms of testing. Also, operation maintenance was hard. We were running the biggest ALM.NET instance in the world, according to Micro Focus. We had the most users, the most data, and the most complex VBA workflow code in a single instance of ALM. This needed to be migrated smoothly to Octane.
In addition, Internet Explorer, which is not the finest tool, was removed from our company's IT at the beginning of 2020. There were a lot of smaller reasons which lead to the need to change to a different tool, to a more flexible tool, to a more powerful tool. For example, the Autonomous Driving guys were going to be adding tons of test cases and automated tests, which would cause ALM, in the configuration we were in, to crash in the future. That was clear, and it was clear to our management as well. We only had a small time window to change to a different system.
How was the initial setup?
For us, it was a very complex setup. It was not only setting up a server, installing Octane, and doing configurations. Our plan was to have a shared Workspace concept with six or seven Workspaces. We did have a major challenge in doing all the configuration stuff, defining methods and processes. We also had to connect at least ten major tools or databases, which are synchronizing information into Octane, or which are used for the special methods of test planning and test automation; pulling information from Octane and running them on our test benches in semi-automated cars.
That was a very complex process. There were some small problems and some bigger problems but we found solutions for all of them.
Because we have some 70 to 80 suppliers that are part of an automated defects exchange, our development, our testers, are reporting defects and those defects are exchanged with those 70 to 80 suppliers. So it's a very complex situation we are in.
Of course, users have to learn different things compared to ALM.Net, the old version of the tool. But we're getting good feedback from the users that as soon as they are used to the idea that they have to use a different tool, they are learning how to use it very fast. There are fewer obstacles in the tool, in this regard, compared to others. Even if you're a JIRA user, you have to overcome that, "I have to use a different tool" issue. There are people who are doing this very easily and with a smile, while some are just trying to stick to the known tools. But that's the change process.
For deployment and maintenance of the tool, there was a major team of experienced IT guys and process guys from our side, about 25 people, supported by about 60 other people just for the special processes of the different development departments. We call them "key users." They are collecting information and reporting it to the core team. For maintenance, it's a team of six people who are implementing changes requested by the core team. Depending on the workload, on average, maintenance is done by three people. There were numerous software developers working on the interface tools, perhaps some 30 IT guys working on the different tools we needed to launch with Octane.
What's my experience with pricing, setup cost, and licensing?
I was not part of the cost negotiations. But the purchasing guys confirmed that Micro Focus offered the best pricing to us.
Which other solutions did I evaluate?
We started analyzing which tool we should use for the future a few years ago, and we started digging into Octane very early.
Options we considered included JIRA, which is used by a lot of small teams. There is a standard toolchain around but with the amount of data and information we need to run, JIRA was very easy to strike from the list. We dug into IBM and PTC Integrity. We looked into ALM Octane vs codeBeamer ALM from Intland. We did some comparisons but ended up with Octane.
With JIRA and the toolchain, you cannot run a business like ours, doing car development, using a lot of plugins to get the needed functionality in one tool. You have so many small companies providing those plugins. What if the company providing one of the tools or plugins you're using collapses or doesn't support a new JIRA version? So this was not an option.
PTC is also using different tools to support the full functionality of requirements, test cases, team management, the backlogs, tests and defects, and so on.
codeBeamer was on the short-list and was the biggest competitor to Octane. But from our perspective, Octane did much better in performance. Octane is not able to do as many relations between the data objects as codeBeamer, but performance was the key factor for us. When you compare Octane to codeBeamer ALM, the UX concept is very good. What's also very important is to have good capabilities for getting reports about the system out of Octane in real time. The reporting engine, the widgets, and the dashboards are a huge plus.
This is still an area where we have lots of feature requests at Micro Focus to enhance things even further. We would like it to be more flexible for the users. We have lots of user-defined fields and lists so users are requesting more capabilities for enhancing this area or would like the possibility of filtering their work items.
We were setting the foundation for the next eight to ten years. We had to have that in mind, as well as the increase in data, the increase in users, the increase in data objects and rules, and the complexity of development which is divided into different pieces. To cover all this in one single tool led us to Octane. Cost and license fees were also a big issue. The two solutions, codeBeamer and Octane, weren't that far apart but, here as well, Octane was first.
What other advice do I have?
Do a quick scan of tools in the market and dig into your needs. Especially for a project with a lot of users with different styles of working together, Octane is the best tool because of the shared space/Workspace concept. Management is able to get a total overview of all the projects or workspaces and the teams are able to operate in their particular styles. That would be my advice.
For small teams, there might be different solutions that are cheaper, JIRA for example, and tools that are more flexible. But if you need to run bigger businesses, Octane is the best because it's replacing a whole toolchain.
The solution can provide a single platform for all automated testing but it's a little different for us because of our strong dependency on hardware, like robots, for automation. We need to have a robot that presses a button, for example, for real end-to-end testing. It depends on the types of errors you're working with. ALM Octane is integrated and fully supporting every task. But on some levels, because of our special needs, we have to work with third-party tools and we then use Octane as a single point of truth for all the results.
Integration of ALM Octane with your CI server is possible and we are working on that so that we can use the features of Octane and connect it to our different departments and solutions. The idea is to try to streamline things and make Octane the central tool for those use cases. Although it's possible already to do this, we have to use some workarounds because of our tools and the way we use the solution. It takes time until the central tools are supporting various processes and, in the meantime, people develop their own processes and their own little tools and they want to stick to their working solution and not start all over again. This is going on in different departments and different areas of the company. So if you then try to integrate all this in a single tool, at the end of the day, you are taking away their "toys" and their "babies" that they invented. So it's a work in progress. But it's possible and it is on our agenda for this year.
The solution hasn't reduced manual testing time in our organization yet, since we are just starting with the integration of our test automation and Octane to create a workflow and process where everything is integrated. This is something we are working on. The first step was to replace the old ALM for a certain number of user groups. We now have more than 7,200 users working in Octane, and we have more than 1,000 concurrent users working in it. It takes time to develop this. But the goal for this year is to integrate it and to use it more and more efficiently. And then it will definitely reduce automated and manual work.
Which deployment model are you using for this solution?