What is our primary use case?
The primary reason that we got Eggplant Digital Automation Intelligence is that we have a couple of systems that are run through virtual machines. Everybody who uses it in our company needs to go through a virtual machine. This solution was the only testing system that was able to run that virtual machine and test within it. It was the best that we found.
What we're mostly testing are web pages and Windows apps; Oracle transportation systems. They could have done local installs, but they put them in VMware and this was the best way to test them.
How has it helped my organization?
In a lot of the systems that I test, I don't have a real way to do full end-to-end testing. I don't have a way to set up data nicely, watch it flow through the system, give expected results and get to a final answer, and then clean everything up. It's very piecemeal. That's not necessarily why we got Eggplant. But what I have been able to do with it are a lot of smoke tests, which we didn't have, and several others. There are also some look-up screens where we set up look-up values for other areas, and we have been able to do full testing on those. We have actually found some errors in one application, errors that have gone into the backlog, and we hope to get those fixed so that we can keep building out more tests. It has enabled true end-to-end testing. So far, we've been able to get Eggplant to test anything that we've wanted to test.
Even with the mobile app, which consumes data from a different system, we'll be able to do some full testing within that, which is great.
Our use cases are somewhat unique, because we're not a software development shop. We are testing the pieces that we can and, for that purpose, Eggplant has helped. It has been great.
The model-based test automation has helped to reduce the test maintenance process. Now, I don't have to do a lot of the simple smoke tests and some other testing.
And while I wouldn't say that the solution has helped to uncover critical bugs that normal testing would have completely missed, it certainly has helped us find bugs and then verify that they've been fixed. In one particular instance, it found out that the continuous integration pipeline was broken, or that somebody forgot to push out all of the correct files, because pages weren't working. It pointed out that there was an issue with the deploy model.
What is most valuable?
I have really come to like Eggplant. Although I'm working in QA now, I worked as a dev for a very long time and I like the solution's scripting language, SenseTalk. It's pretty easy to use.
I also like the IDE and the helpers that it gives when you search something and double-click it. It gives you the outline that you can then fill out in your code, which is very similar to many others. I appreciate that they're doing that.
What I'm also really happy about—and perhaps this isn't technically about functional testing, but it's related to it—is that DAI's newest release allows us to test via scripts rather than models, because we have done 95 percent of our development in functional, not through modeling. I am really happy that then we can use the controller to run scripts rather than having to translate things to models. There are lots of options.
We have mobile apps that we are testing through it with Android Gateway, and that seems to be working great. And, of course, any web apps and Windows applications have been no problem. We are running several scrum lines, and we've just implemented with QA and, therefore, QA is important. What we're trying to do is get the ongoing suite of tests that look at things currently, and then it will be a built-on regression suite as we keep going and keep adding more and more tests. We want to improve quality before handing it off to our customers. If we have mainframe stuff, it's nothing that I've had to test or deal with.
It's been very fast and very easy to use. Because there's that code behind it, we're able to write modularized code, so that when a new page is created, we don't have to rewrite everything, just the things that are specific to those pages.
What needs improvement?
The IDE could be even more full-featured. Because I was a developer, I was very spoiled by either Visual Studio Code or Visual Studio for shortcuts. For example, I was able to say "ctor" and hit Tab and it would create a template of a constructor for me. Or I was able to quickly type out a class mod with properties and methods using prop and hitting Tab. It would set up the template for me. It would be great, when I want to create a new function, if there were shortcut commands like those that help create all of the functions, or if there were shortcut features to do any of the complex plans.
I would also like to see some of the syntax updated. They have the equivalent of a switch, but it's a very weird IF statement syntax. That could definitely be improved.
Another area that I would like them to improve is their database connectivity and ability within a database. Still, we've been able to use it with what they have and get it working.
For how long have I used the solution?
I have been using Eggplant Digital Automation Intelligence for about 14 months.
What do I think about the stability of the solution?
I can't think of any problems. It's been stable for me.
What do I think about the scalability of the solution?
In terms of scalability, since I'm writing functions, it's as fast as I can write them.
About eight months ago, we were going to have a fourth person work on it. They didn't want to get an extra license so we were having to work around the fact that we only had three licenses, but potentially four people working on it. However, given our setup, I could see that it would be easy to add another license if that's what we truly needed and get somebody up and running fairly quickly.
The roles of our users are two QA people and one intern. What they're doing is very similar to what I'm doing. The intern is working closely with me, doing a lot of the tests that I just haven't had time for. On our website we have four sub-domains. I would start the intern on the basics and then she would be able to completely write all of the smoke tests at least, and the basic code tests for that, no problem.
The other QA is the person who is using it through the VM because of the need for the Oracle system. He's also been using that for full integration testing, setting up data, verifying that it goes through, and that it gets the expected result at the end. He's also the one who is working on performance. He wants to start having the equivalent of hundreds of users hitting it to see how well our systems do.
In terms of maintenance of the solution, I handle all the DAI upgrades on the server as needed, but there isn't a lot of maintenance involved. I believe there are three updates a year. It's just a matter of letting everybody know how to redownload and install it. I've updated DAI once and it went very smoothly and worked well. I have a weekly meeting with one of their technical people and for the DAI update I said to them, "Hey, here's my plan," and made sure that it looked good to them, but I was the one who enacted it.
In terms of general runtime and being able to do whatever we need to do, anything that we have wanted to do for our tests, we've been able to get Eggplant to do it for us. We have mainly gotten some smoke and other automated tests going. We want to ramp up with more of those, but the area that we really need to expand into is reporting. Management is going to want some better reports than we've been giving them but we just haven't had time to look at that aspect, because we have wanted to keep going with the tests, first and foremost. We give them a little "okay," or we give them numbers, rather than providing actual reports. We do want to get to that eventually because I know the solution has that capability.
How are customer service and technical support?
We have emailed their tech support and they have been absolutely fine. They have usually responded within a day. If needed, I can jump onto a Microsoft Teams call and work directly with their tech support to get problems solved.
Which solution did I use previously and why did I switch?
Before Eggplant, what we had been doing was C# programming in Visual Studio using Selenium, Gherkin, and Cucumber against Chrome, which is our preferred browser. The reason we went with Eggplant was that we had to use a VM to get to a lot of the products that we use internally, to help test them. The ability to test in virtual machines was huge.
How was the initial setup?
The initial setup was straightforward. It was just a matter of installing the solution locally. It took longer on our side because we had to get a dedicated server, but once we had that server, setting up the license manager on it was very straightforward. Putting in new licenses for it when they expired, until we got our permanent ones, was quite easy. I had no problems with any part of the setup or in running it.
For the third person on the team who's been using it, the biggest issue has been permissions. She's not a developer like the other two are, so she doesn't have local admin permissions. We've just had to go through help desk at times, but otherwise it's been very easy.
Our implementation strategy is that we want to get it up and running and to get scheduled tests and their results. That's what we're working towards. We're hoping to generate some sort of report, whether it's a CSV or something along those lines, to see the daily scheduled test-run results.
What was our ROI?
We're still in the process of setting up ROI metrics. The last number that I saw is that we probably have 200 scripts running. We're trying to get them on a daily schedule, and that is about three hours of testing that we don't have to do, and that's per day. Every day it does the regression testing and we know if we have any problems.
What's my experience with pricing, setup cost, and licensing?
In terms of deployment of the solution, we have an internal server in a private cloud running the licensing machine and we have three dev licenses for the product as well as two run-time licenses.
The development license allows you to run Eggplant Digital Automation Intelligence and allows you to test it and run your script. The runtime license is what is used when you set up a schedule. All it needs is a run-time license to run the executables and run the tests. But since you don't need the full Eggplant Digital Automation Intelligence Functional IDE, you just need the run-time license.
Which other solutions did I evaluate?
The other QA in our organization was looking at a testing product called Ranorex. After they fully understood what it was he was trying to do, they're the ones who said that Eggplant is probably the best solution for that.
What other advice do I have?
Jump in and try it. Don't be afraid of it. Before COVID, Eggplant would train a client for two weeks onsite and then everything would be handled through tech support. Not that the worldwide pandemic was good, but because of COVID we started getting a weekly meeting with one of their tech people, and that has been awesome. He's been a great resource. We go through scripts with him, we can send him something and he can respond back via email. But we also have face time where we can say, "Hey, I tried this, it didn't work," and we can sit there and work together through the issue and figure it out.
Utilize all the resources and play around with it. It's not a simple system, but it's worth learning the details of it because you're going to get so much more out of it.
If someone were to say to me, "We are not comfortable with automating 70 percent of our linear path," that would be fine with me, because I'm not trying to sell anything. But I would point out to them that, by having a robust regression suite that they can run and that they can rely on, they are going to free up their testers to be able to work on the edge cases or the strange business issues. They will be available for all of the manual testing that is either too complex or consists of one or two things that just change often enough that they're difficult to automate. It lets their people then focus on those things rather than just the basics of whether or not the code was released properly or files were forgotten.
The team that I'm on mainly works with web pages. I don't actually have to worry about the stuff that we got Eggplant for. It just so happens that I can also use it to test our web pages. We are not a software development company, we're a transportation company. With a lot of the different tools that we have, what our in-house developers have often done is translate data or move data in file form between databases. Sometimes, the front-end web pages, for example, might end up being a report, or they might verify that customer data has gone in, or they verify fuel prices for comparison to what we paid and to see if we can do better.
Eggplant is not worked into our development cycle. It's a tool that I use to automate tests, but it's not something that we've used in a way that would help accelerate the release of a major enterprise-wide upgrade.
We don't use the solution's AI-driven automated exploratory testing. If we let it loose on our website, it would find a lot of dead ends because if data pages are not updated, and they can't upload that, they can't really go any further. Even if they could upload a couple of files, that requires going off to other systems in the background for testing, and some of it doesn't even go back to a main interface directly. That's not to say that maybe someday we wouldn't work in the model area, but at the moment I personally have been more comfortable and happy to work within the functional model.
Being a developer, the solution has reinforced a lot of my development skills. In the context of the programming acronym SOLID, it is still possible to use programming skills and to make sure that you are writing small snippets of reusable code. It has also shown that QA isn't just manual testing anymore. There's a very big automated component and you really do need people who either want to be developers or are able to develop, because that is what is required now. It's what allows you to make better, more robust tests, and things that can either recover or give you good data on how your systems are performing.
Which version of this solution are you currently using?