Worksoft Certify Reviews

Initially we were automating the regression suite for SAP ECC.

From there we moved into a web application called HVAC Partners, which is something that we've developed that is a type of customer portal. That application also connects to SAP, but it does some other things that don’t necessarily connect to SAP. It is a kind of front end for quotes and sales orders that go into SAP, but it's also reporting status of orders and status of warranty claims and the like for the customers.

From there we moved into the Middle East SAP ECC instance and automated their regression suite and, from there we rolled out S/4HANA for our service business. With S/4, SAP releases updates every quarter. Because the S/4HANA instance is in the public cloud we have two weeks, essentially, to regression test and test any new functionality. We started with the last release and we did about 70 percent of the testing with Worksoft. We also used the S/4 automation tool, which is more for unit testing so it's not as valuable as Worksoft. We're wrapping up that automation in about the next month and we'll be moving on to a European rollout of S/4. We'll just start working our way across Europe and those implementations.

We have it on a virtual server and we're using remote desktop access for the offshore automation engineers to access it.

We're using it with Fiori and it's working fine. We have integration in and out of S/4 to Salesforce.com so we also automated those. The test cases were end-to-end. We start in Salesforce, which is a web application, with, for example, a quote, and then it goes into S/4 and gets reviewed and approved. It then goes back to Salesforce with the approval and a sales order is entered that ends up going back to S/4. And then there's fulfillment, back and forth, and eventually billing and collections. We were able to do that whole automation with Worksoft, plugging into Salesforce as well as integrating to S/4 and doing the S/4 automation, back and forth. It's been incredibly useful. We saved something like 80 percent of the time it would have taken to manually test, using this tool.

In terms of using the Capture feature without knowledge of testing tools, we brought on some new support people. One of them is our web support person and she had no background in Worksoft. She's been using it to do all the initial captures for our HVAC Partners. She's been able to use it very easily. Our more experienced automation engineers will follow up, after she's done the Capture piece, and troubleshoot some of the stuff that she might not understand yet. They're working with her so that she does learn it. But she's been able to use it very easily.

Worksoft’s ability to build tests and reuse them is very good. We ended up obsoleting the tests and not using them with the other tool we used, whereas now, we rerun these, at a minimum, every month. We do that for a few reasons. One reason is to keep the health of the tests up. Suppose a material is obsoleted. The test that has that material in it is going to fail because it's going to say, "Material not found." Or suppose a customer is no longer a customer and he has been blocked or archived. We run the tests to make sure that the scripts don't need any changes. We also use them in case a process has changed. We're releasing changes to SAP about every two weeks: support tickets, enhancements, maintenance, etc. If a business process changes, then the automated test needs to change to reflect that change. Running them every month, at a minimum, helps make sure that everything is healthy.

The other reason is to identify anything in our quality system that could unintentionally impact other things that the programmers didn't realize. We've caught a couple of those in queue and they said, "Okay. I didn't mean to do that. I only meant to change this one thing,” but it changed all kinds of things and we were able to catch that before it went into production. So the reusability is fabulous if you create the tests properly: no hard-coding, and you’re using data tables to hold any of your field selections, and you're using good automation standards, so you create and consume your data. If you create it and consume it, when you rerun it, it does the whole thing again. You don't have to worry about finding a sales order that works, for example. You really have to create a logical test design to make it reusable but as long as you do that, it's very reusable.

It dramatically reduces the time we spend on testing. Before we started using this tool, everyone was pretty much doing testing manually and test events were taking from two to six weeks. What they did in two to six weeks, depending on the scope of what they were doing and how many people they had involved, we can usually do in one to two days.

The most dramatic was when we finished the Middle East automation. They were bringing up another company code and they wanted us to run regression testing on all of their current company codes, about seven of them. We completed it in about four days. The IT director came to us and said that it reduced their labor by 93%. “Quite frankly,” he said, "we would never have been able to do all of that testing. We would have had to engage a minimum of 28 people, and it would've taken them a minimum of eight weeks, and we still would not have been able to do all of the tests. We wouldn't have gotten them done." We were able to do it in a fraction of the time and with a broader scope than they would've been able to do. They would've done as much as they could and then they would have gone live and hoped for the best.

And we've also been able to use it for other things like certain recurring tasks that had been done manually. We had people who were manually monitoring Tidal jobs, which are batch jobs that have been scheduled to run. If a Tidal job fails, somebody has to go in and figure out why it failed and either restart it or fix it, and then rerun it. These are jobs like billing jobs and we automated them. They probably spend 15 minutes a week on billing jobs now, whereas we had somebody doing this about 12 hours a week. And then that person would have to send out an email to whomever the relevant person was saying, "Hey, check your batch job. This isn't running." They now spend about 15 minutes running it. It saves the emails to the users, documents the results in a spreadsheet, and puts it out to a SharePoint where the auditors can pull them any time they want. It was the same thing with monitoring the claims jobs. We've done a few things like that which have added to the value.

Automation using Certify has also saved testing time, big-time. As I said, the Middle East: 93 percent. For the S/4HANA project, what we did in three or four days, they had been taking two weeks and not getting through at all. With the release, you don't get to say to SAP, "Hey, testing is running behind, we need another week," because it's in the public cloud. Like it or not, they're going live. The drill is supposed to be: You test during week one and you remediate in week two and you go live that weekend. We got our stuff done, 70 percent of the work, in about three days, and it was our first time, out-of-the-gate, so it'll go easier with the next release. The rest of the team took the entire two weeks to do their 30 percent. And within the 30 percent they were doing, a lot of them were smaller tests. We were doing end-to-end tests that go through Salesforce and S/4, etc.

In terms of defects, the value is finding the defects prior to moving something into production. There are two I'm thinking of that we found in Mexico. One of them would've brought shipping to a halt and the other one would have brought receiving to a halt. If you shut down factories, even for a short period of time, there is this domino effect. The value of those finds is huge. And this wasn't even something that the guys making this change were testing. They were testing the piece that they changed, which was working. What they didn't realize is that they changed all items instead of just that subset. It was a minor goof in the programming. It was just too broad of a statement. 

I started in IT about nine years ago and we did total manual testing. We would have defects in the high hundreds to 1,000 during the implementation testing. Now, we're probably under 100, so it's much lower. It could be that we're just getting better at implementations.

It's pretty seamless with SAP and Salesforce because they've built in the field definitions and all the things that you need. You literally turn it on and execute your script and it records it. It's very simple. Then you can go back and put in some of the other functions. For example, instead of hard-coding field selections, you put in a data table so you can run it multiple times or with multiple data. It was actually written to work very well with SAP.

Salesforce came a little later. Obviously, big companies like ours don't just use SAP. We have integrations with Salesforce and CRM and CPQ and all these other programs that integrate with SAP. Worksoft started looking at its customer-base and saying, "Okay, what are the popular ones you guys use?" Salesforce was one that came up. So one of their releases, about a year or so ago, included the ability to record in Salesforce like it has for SAP, so it's super-easy.

We’ve used the Capture not only to train people on how to do things, but also to provide the output to our users so they can validate that what we tested was proper. Capture is very good. It is lengthy, though, because it documents every keystroke that you do. At the beginning it will list all the field selections that we use and then it will give you each step: what it is, pass or fail. If we put in a screenshot, that shows up. It's up to you as to whether you put in screenshots or not. A lot of the times the documentation we provide as a PDF is lengthy but it's also very thorough, which is good.

Certify provides codeless, end-to-end process automation across packaged applications. It works well with SAP and Salesforce, for example. Another one that they have done all the definitions for is Oracle. We haven't started on our Oracle ERPs yet, but it's good to know that we can. We just don't have automaters who know Oracle, so we're sticking to what we know right now. The process automation makes for relatively fast automation compared to the other tool that we've tried to use. It makes it so much easier because you don't need any technical programming knowledge. A lot of the other ones are Java-based or based on other tech languages. That's a skillset that the average tester or support person does not have. It makes it very easy for those guys and the learning is quicker too, because the troubleshooting is easier. You look at the code, you can read what it's doing. You understand the business process and you say, "Okay, that's failing because we failed to set this flag, or fill out this field." It’s pretty simple.

I would like the ability to more easily modify the report from the Capture feature. One of the things I don't like is that it keeps repeating all the field selections throughout. To me, if we put them up front, we shouldn't have to repeat them at the different steps. It should just be Pass/Fail and show the screenshot. I've talked to them about this in the past.

There's another part of the Worksoft suite that probably does a better job at documentation for training purposes and providing an understanding of business process. It's the Certify BPP which we're not using right now because we're really focusing on automating all these different ERP systems. Whereas the testing is very detailed, which is great for the auditors and it's great for the users because they see everything we're doing, it makes for some big PDFs. It's a double-edged sword.

Also, with the codeless process automation across packaged applications, once in a while, if we get a weird application that's not widely used, it gets a little stickier. First, the software has to learn the fields, so you have to identify all the fields. Once you do that, as long as there isn’t any non-standard code in the application, then it works fine. But there's that one step that you have to do, a step you don't have to do with SAP and Salesforce, for example.

In addition, Worksoft definitely needs to continue the march toward bringing in more and more of the software that people commonly use. They're doing that, but they can only march so fast.

I know Worksoft is doing some stuff with RPA. There are other tools that strictly do RPA, but aren't automated testing so I'm not sure if they will be able to compete with those. I know that we did do some automation, what we call "bots," with Worksoft, and it was clunkier than some of the RPA tools that are currently on the market. I suspect that they'll come up with a very competitive offering. 

I would also like to see some better reporting of testing status, reporting that we can easily generate to say "Okay, we're 50 percent done and we've got 10 fails and 800 passes." That's what test management software is for and Certify integrates with that. Bang-for-buck, it's probably not a great place for Worksoft to invest. They're probably better off with RPA and bringing on the ability to more easily test software, like Salesforce and CPQ. I'd love to be able to do that as easily as I can with SAP. I would like that same ability to use Capture in CPQ, instead of using Silverlight.

We started using Worksoft Certify in 2016, so it's been about four years.

Stability is very good. They do a lot of releases. They are probably using an Agile methodology, so every time I turn around, they have three more releases out there. It would be helpful if they could release once or twice a year, but I understand why they are doing it. They are adding new features because they want to get them out as quickly as they can. I just don't have time to stop, do an upgrade, and move on.

We haven't had a problem yet with the solution. It's been very good, and you don't have to upgrade every time they do a release. We do it probably once a year.

Scalability is fabulous.

We've certainly taken on more projects. When I first started about nine years ago, there was one major implementation at a time. At the moment we have about six major projects going on and, with the unwinding due to the spinoff, there are probably about 50, but those are not being tested with automated software. We're focusing on just the two SAP ERPs, S/4, and the ancillary web apps. It does allow them to implement faster. Since we did the Middle East, they've brought up two new companies in six months, which is amazing for them. It probably would have been one at a time over a year and a half or two years, otherwise.

We don't use Certify to create RPA at this point. We have so many ERPs to automate that we're sticking to that right now. We're trying to get to where we can pick up more licenses and build up the team so we can start doing some of these other things. Right now, with the spin-off from our parent, everybody is hyper-focused on unwinding. When you're part of a big organization like we were — we're still pretty big but we were huge, Fortune 50 — and you start unwinding things, there are so many shared services and servers that are on their domain, etc. It's going to take us two to three years to unwind all that. So we're marching ahead on our ERPs and I'm keeping my head down. I have my seven licenses, although I want to get about 10 more, but I'm not going to raise my hand until we get unwound.

Most of our issues have been our own internal infrastructure issues. We have a very tightly controlled infrastructure, so I'm always banging up against that. Worksoft has been able to help us solve these problems, and they're not even their problems.

It's far easier than other solutions. We previously had HP Quality Center and we could not maintain it. Prior to my taking over testing, they had implemented that tool. They brought in some outside contractors who did the initial automation and they handed it off to the support team to maintain. But it was so complex to update it when there was an error, or just for general maintenance that needed to be done, that they found it easier to just manually test. They quit using the tool. It was a complete waste.

With Worksoft, in stark contrast, there was a little bit of a learning curve up front because for about 70 percent of your effort you can use its record function that just records your keystrokes. But then you have to go in and harden the script, and put in data tables and screenshots and validations, that type of stuff. But compared to the other tool, there are no real programming skills needed. You learn how to use the functions and when you look at the script or the test, it's not like looking at code. You can actually read it and say, "Oh okay, that's inputting the month and the year," or "That's validating that the sales order posted." It's in English and it's very clear to follow. There's a drag-and-drop, and delete and all the things that you're used to using with other applications, like Word and Excel, that makes it very simple to use. Initially we had a little bit of training involved, but since then it has been incredibly easy compared to the old tool. The old tool didn't make it past a couple of years. It's been four years with Worksoft and we've got interest, globally, from other parts of the company that are asking, "When are you going to automate our regression suite?" So it has been very well received.

Setting it up was pretty straightforward. My biggest frustration was with our infrastructure. We set it up as a remote desktop but our company has all these firewalls and restrictions around access, and my team is mostly offshore contractors.

The offshore contractors have different access than I do. I spent a lot of time whitelisting different web sites to give them the access to the software we are testing.

Deployment took about a month and a half, mostly due to the infrastructure problems. However, now, when we need to upgrade the system, we can pull it down and run the installation. Then, we always get on a call with Worksoft, because if we miss one step and it doesn't work, we can't afford to have the team down. So we get on a call and spend about an hour running through the update.

Worksoft was fabulous help with the setup. They would get on a call anytime. They would help us walk through issues and help us figure them out; even how to navigate our systems. Their assistance during the setup was phenomenal.

We have seen ROI but it's very hard to capture because a lot of the benefits are hard to monetize. We have seen a huge reduction in the time to test and a huge reduction in the number of people needed to test. Rather than lay off a bunch of people, we've chosen to do more projects, so our rate of implementations has gone up. 

The 93 percent reduction in labor that the Middle East calculated was pretty impressive. I would say that, on average, it would be more like a 70 percent reduction in test time, because you still have to have people review the tests to make sure they're comfortable. Even though we say everything passed, they're going to want to review them. And then there's the retesting of any remediation that needs to be done.

The initial investment is probably a little high. It was a little hard for me to sell, but it was a one-shot deal and that's why it's so high. All we are doing now is paying annual maintenance, which we don't have to do if we don't want upgrades, but we do.

It is based on the number of licenses. If we had bought a larger number of licenses, our costs would have come down significantly, which is fair. I did struggle a little bit trying to sell it because our company had already had one failure with a testing solution, and here I was asking for money to try again. However, since we got it in, we have had great success.

We have seven licenses today. The people using it are three automation engineers/quality assurance testers who do SAP ECC. We also have three who do web application testing. They are the ones creating the automation for our portals, e.g., customer portal. I have one test lead who oversees this team and bounces between both SAP and web testing. We haven't bought a whole lot of licenses and haven't rolled it out to a massive number of users. We're doing all the work ourselves.

Since these are concurrent licenses, we could double the number of users with our current licenses because six out of the seven are offshore. While we are sleeping, they're using them.

The solutions we evaluated were all Java-based and they all took skills that we didn't have and we would have had to hire people to use them, or we would have had to train people. The people those solutions might be good for would be developers, but I'm not going to get a budget for a bunch of developers on a test team. And developers don't want to test; developers want to develop. I wouldn't even be able to hang onto these people. That's what failed with our initial attempt. We brought in programmers, they came up with a test, and nobody could maintain them afterwards. It was an investment that we threw out.

There are a number of lessons I have learned from using Certify.

• When you get started with it, you need to make sure that you have an executive sponsor so that you get the cooperation you need.
• Pick up some mentoring services from Worksoft to help you get started.
• You need to document your test cases well. Don't just start without good documentation, because then you make mistakes and then you have to rework that particular test script.
• Be very organized in the naming conventions and the standards you're using to do the automation. For example, don't shortcut. Fill out the fields that explain what the test objective is. That way, when somebody else comes in a year later and they ask, "What does this test do?" it's right there. Be organized.
• Try not to do too much with a single test. We wrote some that were crazy long: 500 to 600 steps because our process was a very complicated process. Step back and think in terms of logical chunks, because a script which is that big is difficult to maintain. You fix one thing and you get 20 percent of the way through and something fails. So you fix that and then you get another 20 percent and something else fails. It will take somebody half a day to fix one script. You can't have that delay when you have 500 that you're maintaining.

I would put Worksoft Certify right up there at a 10 out of 10. It's been the easiest package that we've done. The S/4HANA tool that comes pre-written, where we just go in and change our data to make it applicable to us, is pretty simple but it's not flexible enough. You can only test S/4HANA within those four walls and almost nobody uses just S/4HANA. There are always integrations. So Certify, as a tool that works across integrations from one package to another, documents the results, is easy to maintain, and easy to use, is a 10. I have not seen a package that is this easy and we did look at other ones. This one was just head-and-shoulders above them. It's really a fabulous product, I'm so impressed with it.

We have an SAP environment, so we use Worksoft for SAP and the ecosystem around SAP. Most of the use cases are related to SAP products or interfaces and the applications that are interacting with SAP.

We use it for test automation. We are basically using it for regression testing, especially for our releases. For example, in the big SAP systems, when we have support package upgrades or bigger function releases, we use end-to-end test automation to ensure that the changes are not impacting the processes in the system. With this test effort, we can make sure that the releases are running without any issues in the production systems.

We started using it around six years ago with an on-prem installation, and we had a pretty good experience with that. The way we are using the software is that we have installed it on our terminal server so that not every tester has to install it on his own machine. Having this terminal server environment is allowing us to really stick to specific standards in terms of how the software will be used and in which sequence updates will be distributed on the server. It also helps in terms of the connectivity to the systems that we require for test automation. It makes it quite easy for people to concentrate on developing tests and not on the environment. 

We are running version 12.0, and 2006.77 is the patch level. 

In terms of its web UI testing abilities for testing modern applications, such as SAP Fiori, we started implementing a project two years ago where we developed a logistic layer and a finance layer, which all the future SAP systems of the headquarter divisions will be using. For that project, we had introduced Worksoft for automated testing. We are quite heavily using Worksoft in that area. We have all of our core functionality in that area automated, and we had a really good experience with Fiori. 

Worksoft has these so-called configuration files that you can get for different applications to define the maps. We are also using ServiceNow or Pega for Workforce management. For both applications, you can get so-called configuration files from Worksoft, and with these configuration files, Worksoft can very easily identify the objects. So, you don't need to learn Worksoft from scratch, but you can really build on the foundation of already-existing definitions coming from Worksoft. 

It provides codeless end-to-end process automation across packaged applications. It does not have the approach of writing scripts or having a scripting language for the logic. It is pretty easy to adopt. It is helping us in general because you don't need a developer or a technical guy for building these scripts. People in the business organization can really design their own scripts without heavy IT support. Normally, we just teach testers how to work with Certify in general for a couple of hours. If they have understood the basic patterns in terms of how to find specific commands, how to really work with these conditions, and how to work with varietals, after a day or two, they are able to work with this solution. They might sometimes ask where to find specific things, but because Worksoft also provides master content with a lot of examples, they can deal with it from there. In our company, we have an approach that all people work on the same project. This means that they are also sharing their scripts internally so they can read and steal from others. We also have a concept that for every SAP system, there should be one test architect who is knowledgeable. He is a key user, and he drives the effort to bring knowledge to people. 

It definitely reduces the time you spend on test maintenance. The debug feature, the recognition feature, and the decoupling of scripts and maps are really saving time. Imagine having an error at step 850 in a test script that has 1,000 test steps, and these 850 steps have taken you an hour for execution. In such a case, you have to repeat the entire test because you don't have the possibility to go back to certain steps. Every time, you will lose an hour or two in maintenance. Having these features makes it pretty effective and efficient, but it is hard to say the exact time because you don't know how often your scripts are breaking because of updates. It also depends on the number of scripts. We also have to see the number of saved hours in relation to other tools. So, if you're comparing it with an open-source test automation tool like Selenium, it might be saving you more time, but that might not be the case if you're comparing it with Micro Focus or Tosca. 

It has definitely enabled us to scale up our testing. When you use automated test scripts for test cases, your testers are released from that testing time, and they can concentrate on further testing. The way we are introducing test automation in our organization is that we say, "Okay. This dummy type of testing can be done by a robot such as Certify," and then our testers, who are hopefully more intelligent than the machine, can concentrate more on the individual tests. You cannot really automate all the test cases, and it allows our testers to concentrate on the individual test cases.

Worksoft Certify works well for creating test scripts. As compared to other tools for test automation, what is very good in this tool is the ability to implement logic into the scripts without coding and learning a complex script language. It is comparable to defining formulas in Excel. It is pretty easy to learn how to make your scripts more intelligent and more flexible as per the situation.

The decoupling of the test scripts from the data and the application is also a nice feature. When you are creating test scripts, for example, for a web application, you have to learn about Worksoft and how the controls of a screen can be interpreted by Worksoft. For that purpose, you create so-called maps. These maps are loosely coupled to your scripts, which means if the application is changed, the control will be changed from an identifier. You don't need to rework the entire script. You only need to do these adjustments in the map, and then you can automatically reuse the scripts. So, it is really a smart move to have the decoupling of scripts, maps, and data. 

Its debugging functionality is pretty powerful as compared to other tools. Recognizing the errors sometimes could be challenging. When the debug function, for debugging your scripts, runs on an error, it can stop at that error and identify the elements that may have not been recognized. It can then update the definition to recognize the object. It then repeats the step again so that you have a so-called execution pointer, which you can then use for your debugging. 

Certify is integrated with Solution Manager, but this integration could be easier. 

Overall, in terms of how it is working, I find it pretty clever in all the areas. There are only tiny things. For example, to log into Certify, you have to put in your username and password. In version 12, they changed it, and the password is no longer stored. So, you have to enter it every time you log in. Similarly, there should be a way to store the layout of tables in Certify. You can adjust your tables, but when you close Certify, if I recall correctly, the layout of the table is not stored automatically. So, you have to adjust it every time. I'm, however, not quite certain about it. 

These are tiny things that they can improve, but compared to the whole feature list of Certify, they are not so important.

I have been using this solution for around six years.

We regularly update the software when we see that there are new features available or if there are fixes in certain areas. In general, the Certify software is pretty stable. Based on our experience, there is no need to import patches every month or on very short notice. We normally plan for once a year version update.

In our S4 project, we had the need to develop automated testing for Excel-based solutions. We needed to test the business planning functionality that was running in Excel from SAP. It is quite challenging to build automated test scripts in desktop applications like Excel, but we got quite good support from the offshore team of Worksoft. We had a talk with an engagement manager from Worksoft, and then someone from India came to Lisbon, Portugal, and they all worked together. Our team quite quickly learned how to handle the challenges in that area. So, it is not only about the tool; it is also about the support you are getting from Worksoft. Their support was quite impressive.

It is pretty powerful as compared to other tools. We developed our own tool, and we have also compared it with Micro Focus. We have some knowledge of QTP from HP and Tosca Tricentis. From my perspective, especially when it comes to debugging and also object recognition, Worksoft may be one or two years ahead as compared to the other tools.

When we started with this solution, we had an engagement program. We had a consultant from Worksoft for 20 or 30 days on demand. It was an engagement contract that we had signed while acquiring the licenses. We had two or three onsite sessions. This consultant was here in Berlin with me and helped with the installation and documentation. This engagement really should be seen as enablement. It was not that the consultant did everything and then handed over the documentation. These sessions were more like hands-on sessions, which means our administrators understood how to install the software, how to configure the software, and how to make connections between different applications, especially with the database. They also understood how to make sure that our security regulations are met because there were some problems there. After we had documented everything, the consultant did his job with other clients, and we continued to handle the software on our own. We are deploying patches these days without any support from Worksoft because we simply learned how to do it.

Its initial setup is complex. There is the client part and the database part that you have to install. The client installation is pretty easy and straightforward, and you just have to click the Next button. For the database part, there are SQL Server scripts that need to be executed on the database server. It is pretty simple. You have scripts running on the database, and typically, they run without errors. In all these years, we had problems with the upgrade only twice. We have a QA environment where we typically test the upgrades. We had an error because a column was missing in the table. We raised a ticket, and someone from Worksoft helped us. We learned how to handle it and did the same on the production system without any support. 

If you give me a system, a database server, and maybe a terminal server and we have to install both parts, the database part can be done in one or two hours, which includes preparation time, execution time, and post-installation time. Overall, it would take a day because the database also requires some time for installation. If you are simply differentiating between the effort and the duration, in terms of duration, the database would take a day. In terms of effort, it would take one or two hours. The client part also takes one to two hours, depending on the resource you are using. After that, you only need to do the configuration to connect to the license server and the database. If you know what to do, it would be up and running in a maximum of two hours. We are not really talking about a complex SAP system. It is simply a test automation tool.

We have seen an ROI. In the end, it is money and time. You save time for the testing, and you also save time in making corrections. If you don't have such high-quality testing, you will end up with errors in the production system. You will also have some interruptions in the daily business in your SAP systems. That's one aspect of the return on investment. The easiest way to calculate the ROI is in terms of the effort that you are reducing for testing.

I can only judge based on the situation that we had around six years ago when we did the tool evaluation. Worksoft was not the cheapest, but it provided the value. For 25 concurrent licenses, we paid more than €400,000, so it was not cheap. In the end, if you see how much time you are saving and compare it with others, its price is okay. We had also compared its cost with the licensing costs for HP and Tricentis, and they were at another level.

Now, as we have already booked the licenses, we only have to pay an annual maintenance fee, which is 70%, and that is okay.

The biggest change was not really the tool. There is the saying, "A fool with a tool is still a fool." That's pretty much true. When you are starting with test automation, you basically have to understand the concepts behind test automation, and you have to learn how the robot does the testing. Normally, your testers are reacting, and they are pretty flexible. For example, if they recognize that something is blocking a storage location, they free up the storage location and continue. If you are doing the same with an automated test script, this needs to be implemented in the test script or logic. This is pretty much the difference. So, you need to be very precise in knowing the circumstances or issues that the tool might come across during a test. You also have to have a big focus on the test data. That's because if someone changes your master data, your test scripts will fail, and you won't be able to differentiate whether the error is on the system side or the data side. 

You also need to think about how you are building your end-to-end tests. In the past, most of our tests were in the area of functional tests, but for the dependencies between the different functions, we really had to concentrate on end-to-end testing. This is pretty much the challenge when people from different organizations have to work together. There must be someone from the purchasing team and the finance team to negotiate on specific test cases and test data, which really takes time. With Certify, you have a tool with which you can concentrate on the content and the logic of your end-to-end scripts, and you don't need to spend so much time handling the tool. A good piece of advice for someone who would like to use Certify is that do not concentrate so much on the tool. You should concentrate more on the concepts and circumstances, such as how to ensure the stability of your systems and data. Are you going to introduce a pre-prod system, an isolated system, or an environment? That is more challenging than the tool.

We are using the Capture feature to capture a sequence of our test. Once this sequence is recorded in Capture, we then transfer it to Certify and continue the development there. The Capture feature is kind of a movie that you create. This movie is transferred to the Certify tool, and you can use a feature called BBP to transfer your test scripts into multiple formats. You can transfer it to PDF or Word format. You can show the process documentation with screenshots in a Word document, but in our company, we are very much standardized and formalized. So, this kind of process documentation is not sufficient. We can use it for simple documentation, for example, for discussing change requests for an SAP system, but for comprehensive detailed documentation, we have tools in place.

We have different tools in our company for RPA. RPA is not really in the area of Worksoft. I know that some of the organizations that are using Worksoft Certify for automation are also using it for RPA, but this is more of an exceptional case.

I would rate Worksoft Certify a nine out of 10. I'm pretty confident of and satisfied with this tool, but there is always room for improvement.

We use it to do end-to-end testing for the business. After development has occurred and once we're into verifying that no regression has been broken, it's at that stage of testing that we deploy it.

We are a subsidiary of a larger company and we are focused on rolling it out, at the moment, to our larger company. With the tool's simplicity of use, where we are able to have a code review occurring, in that sense it will be useful in being able to roll it out to the greater company. We will be able to give it to the people who are experts in their areas, rather than trying to pass off test cases to one centralized location. It will be centralized automation and we'll just have one central COE.

Automation using the solution has saved testing time. I couldn't give you a number of hours or days because we're still in the beginning stages of trying to roll it out globally. We haven't been able to use the product and reuse automation. The whole point of automation is that the upfront cost to automate something is heavier and then, as you reuse it, it reduces the testing cycle. We're still investing in the earlier stages where perhaps we have spent equal parts right now, but we intend to see a reduction as we capture more and more.

Certify has also enabled us to find more defects. While I'm focused solely on automating and testing, so I don't have access to the defect count number, I know we have found defects, which tells me that we are finding defects that wouldn't have been found otherwise, or defects that wouldn't have been found as quickly.

• The dataset.
• The reusability.

It does allow for good reusability. When it's designed properly and utilized properly, we can put things in a way that allows for reusability, meaning a lot of reuse of VA01, if they're very similar flows, to keep it simple. And if we do have problems with a more complex flow, we'll make another version of VA01 that targets edge cases.

In terms of web UI testing, we've done very limited Fiori testing, but we have done Salesforce and a few others. Our experience is that when we get that stuff applied properly and working properly, it works very well. They're usually built well and if we do have problems with them we can get Worksoft to fix them. A lot of the times, if we're running on something that doesn't have an XF definition for it, by understanding how it's building objects, we are able to easily map objects fairly well and quickly.

The solution's ability to automate testing for packaged applications like SAP and Salesforce is related to when they do have that XF definition, but I do think it works very well. That's especially true for the SAP integration. That interface is very solid and objects are just about always discovered properly.

Since they updated the Capture feature to a more "Snagit" look and feel, it has become our primary tool. We've moved off of the old LiveTouch functionality. We will use it occasionally, but with Capture being built-in, it's easier for users to be trained on one tool. That tool has enough capability to be able to do both verifying the properties and recording the playback. It works well for us.

Looking at it as a product fully packaged, I would like to see more documentation or ease of use of the documentation. Sometimes documentation does exist but we have to search three different sites to find the proper way to do things or track down the technical document that explains certain fields. 

That, in turn, relates to the ease of use and how objects interact with each other. The application could lend itself to be simpler.

Another area that I would like to see improved is how the permissions are applied. If you're applying permissions groups to a user, one of the options is to delete the group entirely and lose the entire permission group, rather than just deleting the permission from the user, which seems a little silly. In my opinion, that whole module of permissions is very confusing and lends itself to common errors. We have to rebuild permissions occasionally.

The functionality is all there. I just think the way it's packaged can be confusing. We are successful and we can get things working the way they're intended to in Worksoft. It's just that sometimes finding how to do that, or where it is described, can be difficult.

I've been using Worksoft Certify for about the last year-and-a-half.

It's a pretty stable application. When it works, it works well, and it seems to work consistently. And when it doesn't work, it does not work — if that makes sense. When we see it functioning, we've got everything just right, it frequently seems to function solidly. And then, when we seem to have problems, it seems to not function at all, meaning tests will not run, or we cannot get a script to work in this or that particular way at all. But we've been able to work through all of our non-functioning issues through their support.

The solution will enable us to scale up our testing. With our focus being more on regression testing, it increases the testing of existing functionality first, and then we'll bring in that new functionality.

We are planning on rolling this out to more people, multiples of the number we have using it today. We think that it should be scalable but we haven't done it yet on that scale so we don't know for sure. But we do feel it will be scalable and that it will scale well.

Our extent of usage is pretty narrow at the moment. Approximately 10 people are using it right now and they are mainly automation engineers. There are a few directors using it to understand what the product is. People who we would consider to be "automation champions," who will help champion the product at our global headquarters, are being trained on it right now. They're not actually going to use the application, they're just going to understand it so they can help champion it and bring it on, full-scale, with user acceptance. 

Our main users in the future will be those information business analysts who know their respective products very well, the ones who are making the changes in targeted areas and who can easily reach out. They will be able to quickly test and record whatever they need to record for testing. We're looking at anywhere between 20 and 50 additional users within the next year, depending on how well user-acceptance goes, and expansion will continue from there.

I'll start with our positive experience. We always end up with some kind of resolution whenever we do submit something through support. 

There have been times though where their support has been very slow or difficult, where we end up with a level-one support for what feels like much longer than we should have a level-one for the issues we have. These are high-end issues that mean we can't function. That's been a frustration point for us. We've had to meet with Worksoft to talk about the support that we're getting.

As we start to build better in-house knowledge of some of the caveats of Worksoft though, that support has been needed less. That has made things a little better for us and that's why we focus heavily on training and having supporting documents on what we're doing.

I have used Winshuttle as well as DataLoad, which is an open-source and much more simplistic. Winshuttle is used more for something like an RPA function.

Certify has a much deeper bench in terms of what it can actually do. Winshuttle is only functional, to my knowledge, with SAP applications because it's built on the scripting portion of SAP. Its focus isn't for testing, so it's not a good tool for testing. But it is more simplistic in the sense that it looks like a spreadsheet and the result is provided in the last column of what the status bar gave you. It is really designed for one Pcode at a time, in my opinion. Whereas with Certify, you can run a larger-scale test or function or even a larger-scale RPA function, compared to what Winshuttle can support. The complexity involved in that is much harder. It's something of Catch-22, but Certify does enable you to do much more.

I can't speak to about the installation process, as we have a different person who manages installation. As far as setting up users goes, it's fairly simple within the application, once it's installed and functioning on the servers.

We started out with one model of being centralized and we're rotating to a decentralized model of sharing this out with more users and increasing usage. It's almost like we're in a second deployment of the product, and using more of the tools.

We're rolling it out to the specialists in each business area, on the information systems side. These are the people who are producing changes and who understand the changes and updates quite well. We'll have them write the scripts themselves, with our support as the center-of-excellence team. The idea is that they will be submitting the scripts that they've written back to us for code approval and then promotion to gold, to be able to be run regularly, as a script that's been validated. It should work well and be successful for them. We'll give them help with training, etc., in the Worksoft product itself. We're trying to focus on somebody becoming an application expert, for each application we're testing, and to be an application expert for the automation product, allowing them to function well enough within the Worksoft application.

The person who is responsible for installation is also responsible for maintenance of the solution. Like me, he is an automation engineer, but we have different focuses.

My advice would be to develop a very good training program to go with it. Also, understand how to build a good structure to allow for success and to limit exposure where people are editing things that they shouldn't be editing. You should also partner or work with other businesses that have used the solution successful. Build up industry contacts who can help you understand where they're going and where they're having problems, as well, with the model they're implementing.

The biggest lesson I have learned from using Certify is that you can design it to be way more complex than you need to, and you need to be very careful, when you're designing the solution, to design it in a very simplistic manner. It's almost like code in that it enables you to do things that are very complex, but you need to be very cognizant that you shouldn't always do the most complex flow, and that you shouldn't overly design logic out of any one script. They should be relatively simple.

Regarding ease of use, once you understand how to use it you can use it very effectively. But at times it's difficult to understand what the application is doing, what you are actually editing, within the application. So at times, when it comes to certain objects, you might not realize you're editing another object, in a way, unless you've used the application and understand how it actually builds together. It is simple once you know what you're doing, once you understand how all the objects work together, but leading up to that it can be more complex. We overcome that with training, reference documents, and a lot of training documents. We did an intro training with our team just yesterday. We're rolling out more globally, so we're training and trying to have a center of excellence team that can help out with these concepts. For example, they can help design better training to understand, "Hey, when you're editing here, you're doing this." We're trying to do more targeted training to the things we do with our standards inside of Worksoft.

As far as the Capture documentation goes, for us, it's almost too detailed. We've actually implemented a custom solution for documenting, because we need something that's simple, almost like what users would experience for test cases for manual testing. We also designed our own solution for that, in part, because we utilize a lot of Selenium-style code and we need to be able to record results that are occurring in that application. We'll call Selenium and Worksoft and we need to have a consolidated results report. We don't utilize, and, just to be clear, we've never purchased, BPP (Business Process Procedure) so I don't know any of that functionality. But with our unique set up, it did not make sense to utilize those reports. The reporting that is built into Worksoft is good for development cycles, developing scripts, but we don't use it for result-reporting, in the sense of whether the test passed or failed. We've narrowed it down into a custom application.

While it does allow for good reusability, even if best practices are followed, at times it's hard to identify if you have the same components or processes being built. That can be hard to recognize. For example, there will be duplicate login scripts. The application doesn't seem to lend itself to being easy to manage for duplication of processes. We are trying to put workflows in place on our team to help identify duplication and to reduce it. We do intend to use Analyze as a way to help catch duplicate workflows.

We are working towards use of the solution for RPA testing, but our primary charter is to industrialize our testing cycle, and then we can move into something like that.

Our primary use case is for SAP regression testing. We wanted to have our all SAP processes for regression to be optimized using Worksoft Certify. It took around one year for us to take that process and learn how it works, gradually improving it until now. 

We do regression testing on a weekly basis using Worksoft Certify. This does not include just SAP. We are also using the web-based tools when we are integrating end-to-end processes. When we are moving within one end-to-end process, we are shuffling between all the scripts and then going back to SAP. It works quite well.

At some point, we will have an RPA initiative to use this solution. For example, we need to obtain data from our development environment to check reports and verify them, then create transactions using Worksoft Certify.

We have more than 160 end-to-end processes. In our repository, we can have one transaction process or more than 40. There are big end-to-end processes and there are smaller ones. There are processes that can take hours to run and there are processes that can take a couple of minutes to run, but most of the end-to-end processes have multiple transactions. In weekly/monthly releases, we don't have any type of manual testing. We are just relying on Worksoft Certify for its test automation.

We also use Worksoft Certify for test automation in projects.

For scheduling, we are using an Execution Manager from Worksoft Certify.

Our business and IT users have been able to use the Capture feature, even without knowledge of testing tools. We have different IT people from each business area, like production planning or material managements. These people, without even having any knowledge of programming or Certify, just need to know how to use Capture. They are using it because it's really easy to use. You just have to initiate a Capture and do the same work that you are doing every day. Then, Capture will record everything.

You don't need to do any coding when creating end-to-end processes; it's a codeless because it is just recording. We don't need a specific person (or multiple persons) to work with the processes to do coding for the automation, which saves time. It increases the confidence of the process owner or IT person working on it because we provide them the capability to record their own process without doing any coding and just using Capture. They are confident that whatever they have recorded is their own process. It's not someone else recording their processes. The test automator can polish that process, but we are not recording it since they initiated the Capture. Overall, this saves time.

It has reduced our test maintenance time by more than 50 percent because we don't have to do manual test processes. We have saved over 150 man-hours monthly. It has increased our delivery times. We went from 200 man-hours down (three weeks work time frame) to approximately 40 man-hours (three days work time frame).

Using Worksoft Certify, we have found and repaired more bugs, though not that many. Users cannot manipulate information during the process. We schedule our tests at the night time and leave them working. Next morning, when we come into the office, we just check our system. These tests are pass and fail. For the failed test, we have screen captures where we can go and check the screenshot to know what the issue is. No one has to sit in front of the screen when you are running regression tests or scheduling them. I can schedule them and continue with my other daily routine work.

It is really good if you are using Capture in the new version of 11, where you can have a visual of the capture. The new version is much better. If I am using the Capture, I just initiate the capture and start working normally with my transactions. Capture will start recording all the steps that I'm working with. Before sending that one, I can even edit the capture. I can add more steps or remove them. This is one of the best features that Worksoft Certify has.

Worksoft Certify is really easy to use. You don't need any programming skills. If you just know a bit about programming, then you can easily use it because most things record by themselves. 

It's fully capable to automate testing for packaged applications, like SAP and Salesforce. You can do whatever you want and even create one specific process. For this process, you can create multiple processes. For the one transactional process that you have automated, you can utilize that process in multiple other processes too. 

They have different tools for different types of documentation, like Certify and Analyzer (for business processes). In Certify, from the repository, you can get really good, detailed documentation of what has been recorded, how it works, what steps have to be taken, and what data to use for every step. From a documentation point of view, it is valuable and can be used for future training.

Worksoft Certify's ability to build tests is really easy. You can build as many tests as you want with Capture. It is very reusable to apply one process to multiple processes. 

The scripting methodology is easy to learn. It is easy to maintain because it is presented in a simple, narrative way. You don't need to know programming.

When you are using an older version of Worksoft Certify, like version 10, Capture will be running under your status bar. Then, you don't know what transactions you are currently recording. In the old version, you don't have the visual of the capture.

It works well with other tools, but there is some integration required with Solution Manager. If you are using test repository in your solution manager, then there is no direct integration between Solution Manager and Worksoft Certify. 

There is an improvement needed in the reporting within Worksoft Certify. We have either a detailed report or a summary report. We don't have report that can be used for training purposes. A different tool from Worksoft has to be utilized for this.

Worksoft Certify needs a bit of improvement for its web-based processes. It can be difficult because you need to recall the maps, then you still have to add-on for your browser. When you are using the browser-based testing, you cannot even move your mouse or do anything on your system when you are using the web-based testing. Therefore, it needs a bit of improvement on that side. While it does work, it needs improvement. From the SAP side, there is nothing better than Worksoft Certify. However, from the web-based, we are moving towards Fiori. SAP will soon be totally web-based. For Fiori, they need to be great with SAP testing. Thus, Worksoft has to improve the web-based testing part for Certify.

They have a lot of versions coming out every year, like four or five versions. They need to reduce this number. There should be one or two versions every year with add-ons, if necessary. Because if you want to upgrade your Worksoft Certify from an older version to a newer version, you almost need to reset the solution and we don't have that much time. We cannot dedicate four times a year to having the newest version of Worksoft Certify. Though, if I don't need the need changes or improvements, I can skip the most recent upgrade until the next version comes out.

Almost six years. We started launching it in 2014.

It is very stable; nothing happens.

Two people are responsible for maintenance, deployments, and updates: the test automation administrator and the test automator. One is mainly responsible for all the administration and maintenance of the scripts and everything that goes into the integration with SAP Solution Manager. The other guy mostly works with the execution of the script. 

The solution enabled us to scale up testing because we don't need to use manually testing. We are also using it for project testing. 

There are overall 12 to 13 users. We have three users identified for test automation. We have another 10 users who use different packages from models of SAP. They use it to capture their processes.

If you have any issues, cases, or discussion, you can go on the Community portal and ask from there. They have people and technical experts there who can provide you with solutions. It's one of the best things that Worksoft Certify is doing. You can discuss with others at different companies, or possibly the technical experts from Worksoft Certify if we have an issue. Then, we can open a case directly there. 

I would rate the technical support as a nine out of 10. They are very good and available at all the times. The technical support can assist through the Community portal. You can also email them directly. Their response time to contact you back is 24 hours.

Before Worksoft Certify, we needed almost two weeks or more of regression testing to move our changes from regression to production. We were spending more than 10,000 euros for that on top of our man-hours. Even then, it was not proper testing because the user could manipulate things because they were using manual testing. For example, if something was not working, they could try to use some other way to work it. After the implementation of Worksoft Certify, we have increased our scope for regression and have more than 162 processes from all the models. We have also increased the test coverage and have less issues.

We went from two weeks and 200 man-hours to three days of testing. Within three days, we can test for the monthly release. For the weekly release, we need maybe one day or less using Worksoft Certify to test our releases and move them to production. It has really improved the collaboration between teams and our dependencies between various processes. On top of that, it provides really quick reports and results for management. We are capable of providing quicker reports and results for regression, like, "What has happened? What has gone wrong? Is everything okay to move to production?" Management is also very happy with it because it is saving time and money.

I was not involved in the process.

We have seen ROI with its efficient reporting and ability to provide results to management. It has also increased collaboration between different teams, such as, purchasing and sales.

We have used Worksoft Certify as an RPA tool when we have needed to create the master data or upload some documents. This saves a lot of time. It can reduce 10,000 transactions from weeks down to a day and a half.

Purchasing and licensing are okay. Go for the perpetual licenses. In that way, you own a license, then you can purchase maintenance and support on top of that, so you don't have to pay every year for it. Even if you don't want it a contract with Worksoft Certify in the future, you will have your own license of it. Then, if your usage is not that much, you can have one or two perpetual licenses. However, if you want to run your processes, you will need more licenses, e.g., using the run-only licenses. They are really cheap compared to the full licensing.

There are multiple licenses available from Worksoft depending on your requirements. You can take any one of them, and they are really good. For example, if we are using five to 10 perpetual licenses where we need to update or create scripts and run parallelly, simultaneously multiple scripts with just one license, it will take months for us to run 162 processes. Therefore, we have run-only licenses that we use when it comes to executing our test scripts on releases.

It is integrated with SAP. It's one of the best tool to be used if you want to automize your testing for SAP. No other tool is as good as Worksoft Certify.

I have evaluated other tools, but I find Certify really tops them. Most of them are using the command line-based tools. With some of them, you need a expert who knows how to do programming. For Worksoft Certify, you don't need programming skills. For the SAP GUI, I haven't found another tool that can work as efficiently with SAP as Worksoft Certify does.

You just have to be clear about your processes and define them before going to capture one. If you capture some processes, you can just easily manipulate and merge them to the other processes. It is really easy to use Certify, but you do need to be focused. There is also integration with Execution Manager, which can be useful when you are using Worksoft Certify for scheduling and other things.

It is not that easy to work with SAP Fiori as compared to SAP's GUI.

It can be used with non-SAP platforms. 

Overall, the solution is a nine out of 10. For just the SAP GUI, it is a 10 out of 10. For the other tools, like the web part and Fiori, it is about an eight out of 10.

We get user stories from the DevOps teams, in conjunction with a recording they make with another Worksoft tool. Then we will investigate if it has already been automated and, if not, to automate the process which has been delivered to us.

This is all set up in a Citrix environment. We have SAP being used at the moment and we still have the old SAP ECC up and running. I'm not sure which part of this is cloud-based, but the Certify solution is installed on Citrix.

Because of the fact that we started just six months ago and we have a small team. But slowly but surely we are gaining more audience; more people are starting to get interested. That should lead us to be able to start implementing it the way it should be done. We have done some regression testing and, when doing so, we found real issues. So it has proven itself to be useful during regression testing at least.

We have definitely seen savings in testing time. Scripts are executed five or even 10 times faster than any one of us could do by hand. While we don't do so at the moment, we are going to start executing them in a lights-out environment. We will run tests during the night and get more numbers, execute more tests. That should also help us save time. We have to get the experience and the numbers for this, but I think it will save us a lot of time.

The biggest feature is the fact that it's codeless. It takes away the problem of finding people with the correct programming language, since there are multiple such languages. It saves time in introducing people to the solution because they don't need programming knowledge, they just need to be able to think logically. This makes it vastly usable by more people who are not even acquainted with IT at all.

Also, the solution's web UI testing abilities for testing of modern applications is pretty awesome. Like with every product there are some parts which can be improved, but overall it's great.

It's very easy to use and to install. You have to know, as a user, what your exact application is on the test; you need to know which object recognition files you need to use.

You use the tool to do your automated testing. As far as I know at this moment, it can do a lot of stuff. It's usable in DevOps, so with regards to packaged and non-packaged software, it's good.

I use Capture from within Certify. I also have a stand-alone capture that I have up and running. If you look at the whole cycle, it takes the user a lot of time to create the records. During the capture, the responsiveness of the system is really slow. But after that, when you send it to Analyze, the documentation is really easy. Just click the button and choose the format. Automate is the same. You just create automation and choose a file name. Then, when you need it, you just download it into Certify and start using it. We've been doing it for some time now.

The Capture feature helps find the actual processes to test for and to create end-to-end testing. We ask the users, when we are making the recordings, not only to enter the proper data but also to provide us with comments or LiveTouch images of messages that need to be recorded. They know, "Okay, when I see this message then it's up and running." Because they take the end-to-end as a whole in the recording, we can use that as process knowledge as well. So the process is, in fact, being captured in the Analyze software.

In terms of the solution's ability to build tests and reuse them, I would rate it at eight out of 10. We record it on one environment. We make it completely environment-agnostic, data driven. Once recorded, we can reuse it on every single environment in the development cycle, which is awesome.

Performance on the web UI part, especially with some of the more comprehensive Fiori features, like the complex tables that are being used, could be improved. In those cases we have noticed a lot of execution-time increase with regards to the Certify solution.

I have been using Worksoft Certify for six months now. I started using it in September of 2019.

We've had no Certified-related stability problems. We have issues with things like Execution Manager and Analyze. I'm not sure if those problems are infrastructure-related or due to the Worksoft setup, but with regards to Certify it's stable. Sometimes there is a crash, but I think it's more related to the fact we're doing a lot of complex stuff in a Citrix environment with low resources.

The solution hasn't enabled us to scale up testing yet. We're on the way forward, but because of some issues in our own architecture we are not able to execute those tests. But I know how the setup is working, and I think you can scale up really easily; just add more machines, add more users, and have a go.

When I started within the program itself, no one else was using it. There were two users on the Railnova team. At this moment, about 10 or 12 people are using but within a couple of months we will be around 50 users in total.

I would rate Worksoft's technical support somewhere between a six or seven out of 10. In two-thirds of the cases we get a proper technical support member who has the knowledge to help us with our problem. But at other times we get someone who doesn't really know what he or she is doing or doesn't really understand the issue.

Another big part of the grade I gave is the fact that when you are in contact with the call center, a lot of times there really is a lot of background noise. With the accent, it's already really tough for me in fact to understand them, and with the background noise the problem gets bigger. But I've had also a lot of support from the Germans and from all over the world. Most of them are really capable.

It's an easy setup. There are some small configuration settings and then you can have a go. It's all up to the user to do the updates on the definition files. That's also easily available to us.

I started using the software without hearing from Worksoft. I only had to do some courses on the Worksoft University web page they provide. I didn't have any real, proper training, and I was up and running within two or three weeks. And within two months, I was able to provide enough support to get multiple teams within our company up and running with Worksoft. It's really straightforward.

I wasn't part of it, but I believe the initial setup and further configurations took two or three months in total.

Because of the fact that it's also able to do orchestration and because of the fact that our company is moving from the old SAP towards SAP Fiori — they wanted to have the main focus on Fiori for the UI part, in conjunction with the orchestration which Worksoft is able to do — at first it was only UI-driven. But we will expand into more and more Worksoft uses.

The company used a Worksoft consultant for the deployment but I don't know her name.

We, as a company, have good contact with one of the Dutch Worksoft managers and he introduced the integrator to us, as that manager is from the United States. The consultant came over for a week to give us some training on a number of things because we are not only using Certify.

They were really happy with her. A lot of questions were answered, a lot of issues were resolved. She was able to contact Worksoft support really fast. They had a blast while she was here.

We haven't seen ROI yet because we are in start-up mode with Certify. At this time we are only investing in the solution. Hopefully, we will be able to have some insights into ROI within a half-year from now.

The licensing is yearly.

My advice would be to think outside of the box. If you've chosen to work with Worksoft, you have to embrace the product as a whole. You will find, as with any other product, that some things that don't operate the way you want them to or would have expected them to. But if you teach yourself to view your problems from different angles using the software, then you will be able to come up with the most brilliant solutions. You can do much more with a codeless tool than you might think upfront.

The biggest lesson I have learned using comes back to the codeless part. I view myself as a smart guy, but I don't have the proper coding language knowledge. I was working for myself over the course eight years, before working here, and oftentimes the jobs were really cool, but most of the time I had to do Python and this and that. That was always a struggle because sometimes, when you've learned a language but you're not using it for a year or two years and you want to go back, you have to start remembering it. So I was turned down for those jobs. In this case, and we can show the world that it can be done codeless, if you have the proper tools.

When I was first introduced to Worksoft and they told me it was codeless, I was really skeptical. I said, "I don't see that happening," because I had been doing this for quite a while and was used to doing some coding. But the tool convinced me otherwise, which is really nice.

Overall, it's capable of being used in modern technology environments. I have been using it for six months now and I still have a lot of learning to do. And as a company, we need to start using more of the Certify features, not only scripting and rerunning those scripts.

Most of the people who are using it right now in our company already have some testing experience, but it's our goal to have business and IT people use the Capture feature as part of the process for DevOps.

We don't do test maintenance at the moment. We started out with test automation. We had to set up a base for the DevOps teams and then support them from that point onwards. So we are slowly moving into the maintenance part. Because we have split the data from the script itself — everything is data-driven — so it should be fairly easy for us to make the necessary changes. I think execution is faster when compared to human hand movements. But for changing or maintenance, I don't know.

The solution hasn't enabled us to find more defects at the moment, because we have been focusing on "happy path" testing. We need to get to the end-point of the end-to-end testing. But I believe, and I'm rather positive about this, that if defects are entered into the system, given that our regression test set covers a big percentage of the complete solution, it should be able to find defects really fast. Faster than we can.

The Certify users within our company are all in scripting. We're developers. And because we are in a scrum team, we don't have different roles in our team for test automation. A lot of things are being delivered by DevOps the teams, which you can view as functional consultants. As for the deployment and maintenance, a lot of it is outsourced to one of our partners. We do have functional and technical maintenance or support. I'm the technical guy and then we have two functional guys as well.

Its primary function is test automation. We record the test procedures and execute them in an automatic way. It gives us reports, in a centralized way, of all the information that we need to manage test automation for the ERP system and a few other systems as well.

We moved from manual regression testing, once a month, to continuous testing every second day, with the same scope or even an increased scope now. We don't need to focus on the manual regression tests, so we've freed up hands for support activities and new developments. I don't think that it has changed how our organization works directly, but we have become more efficient and more flexible.

In our estimation now, we save about 160 hours of manpower every month because we use Certify. It's 80 hours of robots executing the tests, and they're doing it roughly three times faster than humans. To do the same manually, and keep up with the general pace of development, we would spend 240 hours on testing every month.

In terms of codeless, end-to-end process automation across packaged applications, technically speaking there is some programming or development, like you would learn in middle school. But you don't need to be a professional in any programming language to run it. That's how we've done it. We don't deploy any additional software packages with it. We haven't developed anything else, other than using Certify steps to run end-to-end test automation.

Nevertheless you still have to have a mindset of a developer or an engineer. You need to have understand cycles and other primitive functional blocks. Still marketing-wise, I would call it codeless due to its simplicity. 

SAP is our core and the people involved with it are not always developers. Some of them are analytics or system experts. They don't know how to write C++ or Python or Java. But they can do testing automation in Certify themselves without help from a centre of competence (CoC). The only time CoC will participate is when truly complicated logic required such as runtime variables staking.

In usual situations, the skill set to start doing test automation needed is much lower.

There are a few aspects that are valuable. First of all, there is the screen-grabbing where you can pick up objects and it automatically recognizes them on the screen. Worksoft call it Certify Capture or LiveTouch. It's automatic capturing properties of the UI objects on the screen or on a mobile device for use in test automation scripts.

Other important feature is the modularization of the tests. That's an important feature not only of Certify but of IT tools overall where you create and reuse the components. Our test scripts are done with a "Russian doll" approach. That's standard for modern IT, however sometimes you see IT products without such functionality. Worksoft Certify's ability to build tests and reuse them is done pretty well and balances simplicity with flexibility.

Due to it, the learning curve improved even further allowing non professional test developers to plug-and-play test scripts. Such approach allows you to have that nice, modular end-to-end test. Can it get better? Probably not without sacrificing some other nice features and simplicity.

In terms of the core functionalities important to us there are the actual coding, great test data management, execution and reporting. Those are the basis, which are shared across many test automation tools. 

We also value that the test execution and reporting are stored in a database. We can extract the results and get the data out more easily comparing to text log files. That's different compared to other tools on the market. In Certify, we have a structured test execution and overall structured data.

Last not least is multi interface support is important for our applications landscape. With certify we can test SAP, Net, Java, Web and Android applications with one tool.

Certify's web UI testing abilities for testing of modern applications like SAP Fiori was good when we started and they developed it to be even better. We all know that web applications can change objects in DOM quite fast and it breaks tests. To counter it Certify has made object recognition more flexible and generic, so we don't have any troubles.

For standard and UI stable applications like SAP ECC 6.0 we have no trouble at all due to predefined objects and rich library of methods. That means we don't need to spend time writing a library, which could happen in some of the open source frameworks.

One caveat is that if you start running models in different parts of the end-to-end process — when you really try to hit the sky and make everything automatic, to cover multiple supply chain tiers processes in one e2e test, or similar processes that are really complicated — then tool simplicity turns into disadvantage. On other hand it stops us from unnecessary over engineering the test automation framework.

Architecturally, because Worksoft specifically built a database-oriented application, you essentially store code in the database. Git and text files orientation is more traditional approach with boundless set of tools to control versions, manipulation and analysis. But at the same time, Worksoft supplies us with their own version control inside Certify that has sufficient functionality for now. 

When it comes to mobile testing, we have a small bottleneck there. You have to buy third-party separate licenses if you want to test on a mobile. Business wise we see room for improvement there, although it's that really critical for us. 

None of these are showstoppers for our operations. Worksoft proved to delivery significant improvements in last 3 years and more we wait from 11.5 version. Overall, we are quite well covered with test automation related tools and nothing special is needed.

We've been using Worksoft Certify since 2017.

It's stable. Our approach is that we don't deploy the latest fresh version to production straight away. We wait for a few patches and follow with upgrade instructions. It doesn't matter if it's Certify, DB server, Linux or anything else.

Only case I could recall related to stability problem is a one mess-up in the database during over-few-versions upgrade about three years ago. It required some database cleanup with scripts provided by vendor support.

But looking at the overall stability, it doesn't go down or crash. SLA is 100 percent for us.

Because it is three times faster than manual testing, we can test more with fewer people. That gives us operational scalability already. Platform also has the Execution Manager tool if we would like to have technical scalability via a cluster of virtual machines. In addition Certify itself supports multi-agents, APIs, and integration with Jenkins and other applications including PowerShell. That means we can also develop scalable setup ourselves

As for overall scalability of Certify, licensing is the main bottleneck. You cannot scale more than you have licenses for. You cannot uplift your digital workforce just for one month and run 300 nodes strong cluster for a big upgrade tests. It is not different for majority of IT products, we don't see much of a problem for now.

Certify is our main tool for test automation for ERP, for PLM, and HR. It's covering two major applications that are at the core of any big machinery or manufacturing corporation. We continuously increase the scope, because systems are developed and extended.

We use their enterprise-level support and it pays out. We mostly use it for tricky situations. We often talk with 3^rd level, with architects and other experts behind the certify. It gives us good insights. Separate gratitude goes to active key account management on vendor side.

As for our improvement requests, we speak with the community and we speak with architects and provide necessary information. We don’t know do they take it into their backlog or is it just coincidence that Worksoft have provided most of things we asked for. In any case we get what we require for our testing, we in constant contant and we feel that we are listened to.

We did not have a previous technical solution.

The initial setup was pretty straight forward, in general. There is a database, there are clients to install, and you need a licenses server.

The key thing that you need to know is what you are doing, how you do test automation, in general. That's above the specifics of any tool. You must know how to slice system-under-test into functional components, make the right model for reuse. It is important to break down end-to-end testing into smaller reusable items. That's not trivial, and you faced such challenge with any tool. You have to know what you're doing.

The deployment included one of week training and then a proof of concept for a couple of months. We ran typical end-to-end process in the proof of concept and found our ways to deal with above mentioned slicing, development guidelines and how to establish roles. There are useful guidelines provided by vendor as well. Overall it took about one quarter from the GO decision to start running regression tests.

Our approach to testing is as a "safety net," so we don't need to have end-users who run it and hunt for new defects. It's just continuously testing given scope and raises a red flag if something goes wrong. Such approach secures an immediate feedback for the development team or for the quality assurance team.

End users are not involved into testing usually although we could run hybrid testing with a mix of automatic and exploratory tests.

We didn't work with any third-party. There was initial training from the Worksoft itself for one week. That was it.

The return on investment, when we count the 160 man-hours we save monthly, will take about seven years for our scale

License model is a traditional one. As with any other manufacturing company, the software testing is not our core business, it's not in our interest to invest capital into the licenses. It would be preferable to have a pay-per-use model. At the same time, it's a fair game, for now. 

Aside from their standard licensing fee, if you want to have enterprise support, there's an additional cost.

We have done a few comparisons and we're continuously looking at the market to see if there is anything better, cheaper and more suitable. Market is changing a lot, RPA solutions expanding to test automation market, Robot Framework ecosystem is thriving and some other established vendors are operating. We're keeping an eye on it for evaluation. So far, cost-wise, feature-wise, and in terms of the learning curve or skills requirements, we are good with Certify. From our comparisons now, we would pick Certify again.

As always, you have to know what you're trying to achieve. You also have to think about how do you model your system-under-test; how it is written, the quality requirements and standards. The key benefit comes from reusable test scripts. 

When it comes to Certify itself as a tool, keep your eye on objects. If you map something, keep a good inventory so you understand it and you don't put all the objects in one big basket.

For me, the biggest lesson from using Certify is that you can do quite interesting and complicated things with the codeless approach. You don't really need a complicated programming language to do testing. 

Overall, I would rate Worksoft Certify an 8 out of 10. Of course we would like it to be cheaper and more powerful and developed faster. But are there products that I would rate a 9 or 10? I don’t thinks so. It's at the top of the market from what we know about the market.

The reporting feature is something we look forward to. Also, the Worksoft training that we have done was mainly on the web application along with SAP, SNOW, and Silverlight integrations with the tool. It works efficiently with any tool. The projects we mainly used were for the testing of SAP applications over the web. 

Automating the User Acceptance Testing (UAT) process is the most valuable feature of this tool which helps us save loads of time for our clients on the projects and also generating testing and other reports.

We are doing Worksoft Online Training and also take up various Worksoft automation testing projects. I have totally trained about 50 participants on this tool. Most of the trainees are happy to learn and use this product in their jobs now. We have also done two projects, one for retail sales chain and another for a banking project. Both of the projects were delivered on time and the clients are happy to generate some automated OOB reports and also create their own Receipts. The clients were happy with our timelines and also with the training we offered at the end of the implementation.

Most of the features are good. However, being codeless is the best feature that makes it easy to use for those who have done any coding but understand simple English like commands. Anyone with basic knowledge of using a basic tool can learn and implement the solutions with Worksoft. This is why we have been able to train so many consultants on this tool and they use the same in their jobs currently and keep referring us for more candidates. We recommend using this tool for your automation testing projects.

Pricing is a bit high and we would like to have the availability of a trail environment for beginners and training would be great to have and easier to expand and use by more and more consultants.

The bigger reports needs to be more customized by the tool for better use, also tool also can be further simplified by a better Graphical user interface (GUI) will help us a great deal while taking up projects. 

We look forward for these upgrades so we can enjoy using the tool more and help others learn this tool.

I have been using this for the past three years. It has evolved as a codeless automation tool for testing. We used sed 10.0.1 and are currently using 11. 

We have also used few other tools for our trainings. 

ROI is pretty good so far.

We would like to have a cheaper version for a single user/trail version of the same for our trainings. The cloud based environment should be available free for the trainings for 15 days or so.

We do trainings on a needed basis so we understand the demand of various tools in the market. We strongly believe while Worksoft is a good tool it is losing its potential a bit to other tools in the market. The demand for training has gone down in the last two years.

The reporting is good.

It is very user-friendly.

Certify integrates with other tools and it works very well with other machine testing applications.

If we can import process from other automation tools like UiPath, Selenium, or Automation Anywhere, then it would be more helpful.

I have been using Worksoft Certify for more than seven years.

This is a stable product and we used it on a daily basis.

This is a scalable solution.

We have created tickets in the past and the support is absolutely fine. Their responses are very fast.

We did not regularly use a similar solution prior to Worksoft Certify, although I have worked with others here and there.

The installation is simple, taking less than an hour to complete.

Our internal IT team performed the installation and there is no maintenance required.

I have been comparing other tools, such as Micro Focus UFT One, to see whether they are feasible for our purpose.

Overall, this is a very good tool and I recommend it.

I would rate this solution a ten out of ten.