What is our primary use case?
As we began architecting a CI/CD pipeline in our SDLC, a need quickly arose for left-shifted, continuous testing of our web applications in real browsers. Specifically, a service that could integrate seamlessly with our Jenkins CI servers and executing tests continuously against our Dockerized applications and firewalled environments. Furthermore, our company does not have the resources to implement and maintain a proprietary Selenium Grid for test distribution. Sauce Labs addresses all of these concerns, and furthermore offers a platform for manually debugging applications in any combination of platforms, browsers, and devices needed, either from an automated test script or from a manual test session. The largest selling point for our needs is its relatively seamless integration with our Jenkins CI servers, including detailed playback and test reports per build job, as well as easy configuration of the proxy tunnel used to access our firewalled environments and desired browsers-under-test, right from the Jenkins UI via their Sauce OnDemand Jenkins Plugin.
How has it helped my organization?
Prior to moving to a continuous delivery model, we were on a classic two week regression testing cycle, mostly conducted via manual testing and manually-executed automated tests scripts in a large integrated testing environment. Upon shifting our SDLC to be more continuous, and rearchitecting our delivery pipeline, we were able to very quickly shift that long testing cycle to be integrated into the delivery pipeline itself by left-shifting these deep functional/regression tests to be executed during build time on our CI servers, and the implementation of these automated tests scripts being integrated into the development effort itself. Sauce Labs allowed all of this to happen by supplying the service needed to execute automated browser tests in from a CI server without having to implement or maintain our own proprietary grid of virtual machines.
What is most valuable?
Of all the services explored, Sauce Labs offers the single best solution for integrating deep automated browser testing in a CI/CD pipeline, and integrates best with Jenkins. The Jenkins UI plugin offers a way to easily configure browser settings for tests being executed continuously. The Sauce Connect Proxy is a necessary technology for companies who have firewalled testing environments, as well as localhost testing. Test reports (that also integrate seamlessly with Jenkins jobs) are also detailed and incredibly useful. Browser support is quick to be updated as new versions of browsers are released. The custom capabilities that can be provided to Sauce Labs VMs during automated testing sessions are a valuable option for experimental or niche testing.
What needs improvement?
The Jenkins Sauce OnDemand plugin could have more options available to tap into more of the custom capabilities Sauce Labs actually supports. Currently, in order to tap into these capabilities, a programmatic solution is required from the test code rather than being able to configure them via the Jenkins Plugin UI, e.g., desktop resolution, browser automation binary versions, remote Selenium version, and browser console logs. These are all things it would be nice to be able to customize straight from the Jenkins Plugin UI.
In our experience, we have also had some issues with the Sauce Connect Proxy on our Jenkins servers failing to start, which makes the optimal CI/CD pipeline come to a halt.
User account management needs an overhauls, allowing for user groups rather than just a hierarchy structure. Test job queuing is a first-come-first-serve, meaning there is no inbuilt way to queue batches of jobs from different Jenkins servers/accounts.
For how long have I used the solution?
One to three years.
What do I think about the stability of the solution?
Occasionally there are issues with the stability of the Sauce Connect Proxy during Jenkins builds, which can cause builds to fail seemingly inexplicably, since determining the root cause is oftentimes difficult.
What do I think about the scalability of the solution?
The only issue is cost. Each new unit of concurrency you add to your enterprise license represents a relatively linear increase in cost.
What's my experience with pricing, setup cost, and licensing?
When you reach the Enterprise licensing tier, base level being with 10 concurrent test sessions, pricing is essentially per-unit-of-concurrency thereafter with a relatively linear increase and not much benefit for "bulk".
Which other solutions did I evaluate?
BrowserStack was also explored, but did not integrate with out intended CI/CD model as extensively or seamlessly as Sauce Labs was able to.