What is most valuable?
There are a few features that makes this service a top-tier solution. First there is the automatic video recording of the test runs that records the actions taken. For any debugging that you have to do this makes this service invaluable. Often times I would look at the recording first before I started debugging the test locally since I would likely find the issue there faster than starting up my own debugging session.
Second best feature is the use of real mobile devices as an option for some of the test runs. There is no substitute for running tests on real devices. After setting up and maintaining a local testing grid of 9 mobile devices I can assure you using a service that does it for you is so much better. There were some weeks that were spent on just fixing issues when using my own devices.
The best feature though is the fact that they use the WebDriver protocol for executing actions on their service. A lot of companies just used their own variations on the WebDriver. This then meant that you had to write another set of wrapper functions just to make the current framework work with whatever vendor you were using. Sauce Labs just used the standard remote WebDriver. Which meant that all you needed to change was the hub url to point to their hub and you were done. If you wanted to there is a few lines of code that let you mark the test runs as successful or failed from your side instead of their generic completed message. All told that is 5 minutes of work and it would have better integration with whatever your process is.
How has it helped my organization?
The ability to share the tests that failed with the developers improved the overall efficiency of the team since there was no ambiguity on why a test failed. In addition we used some of the recorded test runs with the business to clarify the functionality of some of the more complicated features.
What needs improvement?
My company wanted test coverage for some devices that were not available on Sauce Labs so I still had to manage a small "test lab" with those devices. Some of the devices were Windows Phone 8 and Blackberry 7 and 10. On our site the only reported instances of those devices accessing our site was the test devices after 6 months so I am still unsure why we wanted to support those.
What do I think about the stability of the solution?
Actually since their service was so stable it helped identify some flaky tests. Before those tests were executed on bad machines so if they failed they would be executed again and then it would pass. But by running the tests on Sauce helped identified that the test itself was flaky so the tests were refactored into stability.
What do I think about the scalability of the solution?
We did not encounter any issues with scalability. I personally wished that we had a budget to get more parallel executions but the cost was not justifiable.
How are customer service and technical support?
Their customer service reps are fantastic! They have heard of all of the typical issues that can occur in business and offer solutions to it such as using one of the professional tiers to make sure that their service integrates into the workflow before going to the full blown enterprise plans.
Which solution did I use previously and why did I switch?
We had used Perfecto Mobile but was unhappy since it was not easy to integrate with our existing framework. Business wanted the devices to be real devices everywhere in the world so went ahead and purchased Perfecto Mobile without trying the service.
How was the initial setup?
The initial setup was so easy that I thought I must be missing something. Since their service provides so many features I was certain that there must be some other options that I have to declare in the code to get those features, but no it was really that straight forward.
What about the implementation team?
I implemented it in-house since I was tired of managing 25 virtual machines and 12 mobile devices and 5 physical machines on a seemingly daily basis.
What's my experience with pricing, setup cost, and licensing?
Truly agile companies that are constantly testing should get the enterprise solution that they have with dedicated machines. For more ad hoc testing or just testing before the release the pricing on their site would be the best.
Which other solutions did I evaluate?
I evaluated Perfecto Mobile and BrowserStack before testing out Sauce Labs and they did not compete well enough with Sauce Labs. Perfecto did have more devices available around the world than Sauce Labs but the connections were slow.
Disclosure: I am a real user, and this review is based on my own experience and opinions.
Jul 17 2016