The most valuable feature would have to be the cross-browser testing functionality. Our product specifications say that we support the Big Four browsers – Chrome, Firefox, IE, Safari – latest versions and up to two versions back. When including mobile, that’s 10 different browsers (and browser drivers) to maintain. This would be a near impossible task for our small testing team.
It also gives us the ability to replay our tests back in video form. This makes attempting to debug failed tests not a breeze, but easier. It also helps convincing (stubborn) developers that there is a problem.
How has it helped my organization?
We have been able to successfully integrate Sauce Labs into our automated test environment. From that we are able to run our test suite across multiple browsers overnight. This gives us a level of confidence that we previously didn’t have.
The other huge benefit we get from this product is the manual testing powers. When a client tells us that they’re having issues on a particular browser. We’re able to fire up that exact version and reproduce the problem ourselves.
What needs improvement?
Speed is the key. We’re a business in Australia, so any tests that we perform are having to travel to America and back. This is time that we have to wait on for us to get responses back. In an active business every second counts.
For how long have I used the solution?
We've been using it for about 12 months.
What was my experience with deployment of the solution?
We've had no issues with deployment.
What do I think about the stability of the solution?
We've had no issues with instability.
What do I think about the scalability of the solution?
We haven’t had any problems that we couldn’t work around, or there wasn’t any documentation on. The only problem is that hit that point where it’s unfeasible to expand our UI testing framework as the cost to run a shed load of tests is too much for our business.
How are customer service and technical support?
This has been an area of focus from Sauce Labs. Their customer service is unmatched, and their help documents have all been recently updated.
Which solution did I use previously and why did I switch?
We have used a local Selenium grid, and still do to write our tests. However, we valued the cross-browser accessibility over speed for our nightly automated test suite.
How was the initial setup?
Initial setup was somewhat difficult originally. That was due to multiple reasons, namely lack of actual programming experience on our behalf, and the documentation at the time was a mountain to climb. Sauce Labs has since updated this documentation, and we have since gained experience.
What about the implementation team?
Our in-house team performed the implementation. For advice, it’s a lot like setting up Selenium Grid. One thing we did was have an Environmental Variable “ISTESTLOCAL”, we use this as a gate to whether or not our tests should run on Sauce Labs or not. This is very helpful in increasing the speed which we can write and debug our own tests.
Also, if using the Bamboo plugin, make sure to check the settings page as there is a list of already in use environmental variables set when Sauce Labs runs.
What was our ROI?
The biggest things you have to consider is how much testing you’re planning on doing, who is your client base, and what browsers they’re using, and what can’t be tested using API testing.
What other advice do I have?
If you have any issues with the product just shoot the support team an email. They always get back to me promptly with the best solution they have available. Also, spend the time to read the documentation.
If your company is security conscious, as it should be, look into Sauce Connect which creates a secure tunnel between Sauce Labs and yourself.
**Disclosure: I am a real user, and this review is based on my own experience and opinions.
Quality Assurance Engineer at a tech services company with 501-1,000 employees
Supplying devices to a testing team of possibly close to one thousand testers and developers is a great undertaking but Sauce Labs has made this very easy and a welcomed solution.
As stated earlier we use Sauce Labs for a combination of automated testing and manual testing. Therefore the most useful features are the ability to run the functional automated tests via a Sauce Labs tunnels which allows access to applications in our internal network. The second most useful feature is the manual side.
The most valuable feature is the cross-browser feature, it has many android and iOS devices both simulators and real devices. It's easy to integrate. I also like video recording too.
Sauce Lab analytics helped us to get detailed knowledge on test cases execution and logs.
I have found the live test section with Sauce Labs to be extremely valuable. When you can't quite figure out why a test is failing, you can go to the live test results section within their tool and launch your test (specifying a given OS/browser, or device) manually and step through the test to see the issue more clearly, usually opening up the developer's tool console and watching the network calls and console (within Chrome) to usually find the underlying issue.
The insights section provides a great overall state of the automation suite and can identify trends relatively quickly. If we see a dip in our passing rate over time, we can look at what changed when the test started failing to find the root cause rather than doing a quick fix to find that the test fails a short time later.
Live device testing. As we all know, It's really hard and challenging to find/purchase many real devices to test because it will be costly and not all the team can be able to purchase all of the devices out there. We used to have a lot of real devices under our labs. However, it is really time-consuming to maintain those devices and make sure they are up to date with the testing requirements.