What is our primary use case?
Our CI/CD (Continuous Integration / Continuous Deployment) QA Pipeline interacts with Sauce Labs when it goes to run the necessary automated tests. Our automated tests exist for normal web browser tests (we specify to run on the latest Chrome version, on Mac OS) as well as mobile tests where are mobile tests run on various Android and iOS devices to verify that our apps are working properly on multiple systems. We use a 30 VM (Virtual Machine) farm from Sauce Labs, which gives us enough VM's to get through all our tests in a very reasonable amount of time.
How has it helped my organization?
I use it to maintain my own VM (Virtual Machine) farm of 25 VMs. The admin time I had to spend on maintenance and upkeep was a few hours per week, coupled with the fact that the up-time on the VMs was probably somewhere around 75% due to issues coming up in off-hours when I wasn't available to due maintenance/fixes/repairs. When I finally moved over to Sauce Labs, all this burden of doing the management was uplifted off my shoulders, and the up-time is nearly 100% for Sauce Labs VMs. On top of that, we can specify from an abundance of device types, OS's, browser versions, etc. when running our tests. Furthermore, the analytics side of things within Sauce Labs has drastically improved over the years and you can dive into your test results (pass/fail) to get a better understanding of how things are working over a period of time. Overall, Sauce Labs has provided us with a VM farm that is virtually always up and running and has taken the burden off of us/myself of maintaining the VM farm.
What is most valuable?
I have found the live test section with Sauce Labs to be extremely valuable. When you can't quite figure out why a test is failing, you can go to the live test results section within their tool and launch your test (specifying a given OS/browser, or device) manually and step through the test to see the issue more clearly, usually opening up the developer's tool console and watching the network calls and console (within Chrome) to usually find the underlying issue.
I also frequently use the insights section within the tool, to analyze trends of the pass/fail rate of our tests, so I can see what suites of ours are more failure-prone than others, etc. The tool allows you to filter up to the last 30 days' worth of data, along with many other filter options (OS type, tag, build, etc.) so you can really produce insights into your tests.
What needs improvement?
Overall, I think Sauce Labs provides us with a valuable tool and resource. As far as what could be improved, I would say the overall test execution time. Some of the calls take a bit longer than I expect, for example in web browser tests; while the execution time isn't obnoxious, it could be improved so that overall tests/test suites finish faster.
Within the insights section of the Sauce Labs tool, I think it would be nice if you could go back farther than 30 days worth of data. I like all the custom filters available, but being able to go back say 90 days (or more) would show more statistics on our data/test results.
In the next release, I'd like to see continual new additions of OS's. This area is usually pretty up-to-date, please keep at it!
For how long have I used the solution?
What do I think about the stability of the solution?
Very good, up-time is near 100%.
What do I think about the scalability of the solution?
Very good, you can add Vs if necessary.
How are customer service and technical support?
We've only usually contacted customer service (our account manager) to add VMs, it's been seamless and pleasant.
Which solution did I use previously and why did I switch?
Manually maintained my own VM farm. We switched to Sauce Labs as the cost/time to maintain my own VM farm was too high.
How was the initial setup?
Straightforward and easy, with good instructions in their knowledge base.
What about the implementation team?
What was our ROI?
I'd say it definitely provides a good ROI, vs maintaining your own VM farm.
What's my experience with pricing, setup cost, and licensing?
The cost is worth it. Start with a VM farm that you think suits your needs, and then add/subtract VM's if necessary.
Which deployment model are you using for this solution?
If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?
Amazon Web Services (AWS)