Andy Antes - PeerSpot reviewer
Senior Software Engineer at a computer software company with 10,001+ employees
Real User
Saves us time, good support, and helps us ensure product quality cross-platform
Pros and Cons
  • "The most valuable feature is the ability to run concurrent automated tests up to a specified value, depending on what we are currently paying for."
  • "This product is not super scalable, because you have a very specific number of VMs that you can use."

What is our primary use case?

We use the Sauce Labs test automation VMs to help our quality assurance engineers run automated tests concurrently, using a platform that we build in-house.

We use the browser VMs and we definitely use it for end-to-end testing. We may use it for performance testing, as well.

The way that we use it is pretty straightforward.

How has it helped my organization?

This product is a big time-saver, so by extension, it's a cost-savings.

It's helped us remove the need for extensive manual testing, which is how it's freed up a lot of time for quality assurance engineers to work on other things. They can work on the automation itself, for example, whereas otherwise, they'd be spending all of that time doing these tests manually and synchronously.

Sauce Labs is optimized for automation and intention with major CI/CD platforms and developer tools. In our case, we've plugged it into our pipelines quite easily. It's an integral tool in that regard, so it is very important to us that it is optimized for integration.

We run Sauce Labs with almost every code commit.

Although Sauce Labs is a cloud-based solution, latency has not been a concern for us. We haven't run into any situations where latency is causing any major issues.

What is most valuable?

The most valuable feature is the ability to run concurrent automated tests up to a specified value, depending on what we are currently paying for.

I am pretty satisfied with the options in terms of the number of browser and OS combinations they support, as well as the number of real devices, mobile emulators, and simulators. I haven't had any needs that they couldn't satisfy. This enables us to ensure the quality of our products, cross-platform.

What needs improvement?

A really nice feature would be a pay-as-you-use model, as opposed to a strict number of VMs that you pay for continuously.

Buyer's Guide
Sauce Labs
April 2024
Learn what your peers think about Sauce Labs. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
771,170 professionals have used our research since 2012.

For how long have I used the solution?

I have been working with Sauce Labs for two years and nine months. It has been used by the company since before that.

What do I think about the scalability of the solution?

This product is not super scalable, because you have a very specific number of VMs that you can use. If there was a pay-as-you-use-it model, I think that would really increase the scalability of the solution, but right now Sauce Labs is a bottleneck because you have a specified number of VMs.

We use, on average, 75 of their VMs per hour. Optimally, we're using as much of the VMs as we pay for. As such, the goal is to have our tests running often enough to maximize that usage.

I can't speak to how widespread this product is used in the company, although in my team, there are 10 of us using it. My team is serving between 30 and 40 quality assurance engineers. Other teams in the company are also using it.

How are customer service and support?

I have not personally used Sauce Labs' technical expertise to assist us with integration or for implementing solutions, although, in the past few years, I've submitted a couple of trouble tickets based on some technical difficulties.

I found the technical support to be great and timely. I haven't had any complaints, and the issues have been resolved.

Which solution did I use previously and why did I switch?

I don't have any experience with other similar technologies.

How was the initial setup?

I was not with the company when it was first implemented.

What's my experience with pricing, setup cost, and licensing?

We pay for a specific number of VMs. The licensing model should be changed to pay-as-you-use, which would make it more effective.

What other advice do I have?

The biggest lesson that I have learned from using Sauce Labs is that concurrency and automation save a lot more time than you initially think.

I would rate this solution a nine out of ten.

Which deployment model are you using for this solution?

Public Cloud
Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
PeerSpot user
reviewer1400664 - PeerSpot reviewer
QA Analyst II at Regal Entertainment Group
Real User
Enables us to configure multiple test scenarios and pinpoint which internet browser we want to test
Pros and Cons
    • "Another feature that could still be improved on is more error clarity. Sometimes when running automated scripts the test will fail on the device side instead of the script and errors only show a 500 try again message instead of a detailed script that could of a been a timeout error from the code."

    What is our primary use case?

    The primary use case for this solution is for automation testing on mobile and web on our testing environment so we can implement continuous integration and continuous delivery into the workplace. With the automation practice being used we are able to deliver more and possibly deliver daily after each build is created in theory. With the automation, we can now focus time testing the higher traffic areas or higher risk areas that could possibly crash or cause a bad experience for the end-user of either the web or mobile application. 

    How has it helped my organization?

    Sauce Labs has improved the organization with the ability to test more internet browser configuration, more mobile testing devices that are not a flagship phone, and more operating systems. With the internet browser testing, we are able to config multiple test scenarios and pinpoint which internet browser we want to test on and what operating system we want to run the test on. So with Sauce Labs, we are able to cover multiple Chrome, Firefox, and safari variations to support and test for all users. 

    What is most valuable?

    The feature that I have found most valuable is their device bank. With this, I am able to test more niche phones that is not a flagship and have the ability to support the end-user. This has been useful because when older devices from Android and iOS are running older OS and they experience a crash, it has been hard in the past to replicate their issues without purchasing that device and hoping it matches the phone specs. With the device bank, it seems like all a major of phones from android 5.0 are covered up to the current. 

    What needs improvement?

    A service that could be improved is some of their Wiki documents are either outdated or point to broken URLs; however when contacting the customer service they are able to respond in a quick manner with either an updated document or with the steps to fix the issue. 

    Another feature that could still be improved on is more error clarity. Sometimes when running automated scripts the test will fail on the device side instead of the script and errors only show a 500 try again message instead of a detailed script that could of a been a timeout error from the code. 

    For how long have I used the solution?

    The company has been using Sauce Labs for 8 months.

    Which solution did I use previously and why did I switch?

    No we were not using a different solution. 

    What's my experience with pricing, setup cost, and licensing?

    I think a longer trial period would be beneficial. Instead of 14 days look at 28 days due to in part the company that is on the trial period could be busy or possibly busy during the trial period. 

    Which other solutions did I evaluate?

    We looked at other options but found that Sauce Labs was the best option to support both Web and Mobile.

    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    Buyer's Guide
    Sauce Labs
    April 2024
    Learn what your peers think about Sauce Labs. Get advice and tips from experienced pros sharing their opinions. Updated: April 2024.
    771,170 professionals have used our research since 2012.
    reviewer1396401 - PeerSpot reviewer
    Senior QA Engineer at Indeed.com
    Real User
    Increases cross-browser testing and reduces cost in purchasing and managing devices
    Pros and Cons
    • "Sauce Lab analytics helped us to get detailed knowledge on test cases execution and logs."
    • "They should provide a JIRA integration plugin so that we can easily log issues."

    What is our primary use case?

    It is usually used for manual and automation testing for different browsers and OS.

    Uses:

    1. Running image comparisons on different devices

    2. Run test cases on mobile and desktop OS and browsers

    3. Running test cases on both production and QA

    4. Sauce proxy helps us to run tests locally

    5. Sauce lab analytics provide better experience to analyze failures and get run timing of test cases

    6. Using it for both manual and automation

    7. Can help to do compatibility testing of code in different OS

    8. Reduce manual overhead

    How has it helped my organization?

    • It increases cross-browser testing and reduces cost in purchasing and managing devices
    • Helped team to run test cases locally on OS and browsers
    • Reduce device cost and handling, We lost many devices earlier
    • Ran test cases through Jenkins and capture results in a cool format
    • Easy to debug issues in all supported devices
    • Running test cases on both production and QA
    • Running image comparisons on different devices
    • Sauce l=Lab analytics provide a better experience to analyze failures and get run timing of test cases 

    What is most valuable?

    1. For automation, it is really helpful as it provides logs and videos of test cases executed

    2.  Parallel test execution helped us a lot to reduce test time

    3. No maintenance of real devices

    4. Sauce lab analytics: It helped us to get detailed knowledge on test cases execution and logs

    5. Sauce proxy: Helped to run test cases on an internal network

    6. Very easy to enable for automation

    7. Vast devices and OS combinations helped us to keep track of world trends and usages

    Helped us in testing IE devices is really beneficial

    What needs improvement?

    Improve easy way to automation testing and generate script automatically while running manual test cases on Sauce. 

    1. When running manual test cases, provide us a way to record and playback the same steps

    2. Provide JIRA integration plugin so that we can easily log issues.

    3. Provide easy way to create scripts automatically for manual steps and use sauce labs UI to execute it again

    4. Sauce labs should provide their own email client and easy to send email in that user session

    5. One-click setup for sauce proxy

    For how long have I used the solution?

    I am using it for the last four years.

    What's my experience with pricing, setup cost, and licensing?

    It is really cheap and fast.

    Which deployment model are you using for this solution?

    On-premises
    Disclosure: I am a real user, and this review is based on my own experience and opinions.
    PeerSpot user
    Software Engineering Manager at a healthcare company with 10,001+ employees
    Real User
    All-in-one testing site with endless functionality
    Pros and Cons
    • "Since this is an all-in-one testing site, we are able to take advantage of the browser OS combinations, mobile emulators and simulators, and real mobile devices. This is important to us since we have a variety of users, browsers, OS, etc."
    • "I would like for there to be more detail in regards to the quality of our code i.e. how many failures occurred, how many passed based on industry standard metrics, etc."

    What is our primary use case?

    Our healthcare company has a QA team who uses Sause Labs to review various metrics.

    For me, as a software engineering manager, I am able to see who from my team is logging on, when are they logging on, and other things in regards to activity within the environment.

    How has it helped my organization?

    The platform is optimized for automation and integration with major CICD platforms and developer tools. This allows us to speed up our production without burning up all of our resources.

    It also provides access to automated functional testing (this couples nicely with our other functional testing that we have), visual registration testing (which is as easy as a click of a button), and front end performance testing (which provides a clear understanding of our current state of performance).

    Sauce Labs provides us with an all-in-one solution that saves us time, money, and human capital.

    What is most valuable?

    Since this is an all-in-one testing site, we are able to take advantage of the browser OS combinations, mobile emulators and simulators, and real mobile devices. This is important to us since we have a variety of users, browsers, OS, etc.

    What needs improvement?

    I would like for there to be more detail in regards to the quality of our code i.e. how many failures occurred, how many passed based on industry standard metrics, etc.

    For how long have I used the solution?

    I have been using this solution for a few months.

    What do I think about the stability of the solution?

    This is a stable solution, I have not had any stability issues.

    What do I think about the scalability of the solution?

    As told by our rep, this is a fairly scalable solution.

    How was the initial setup?

    I was not a part of the initial setup though I have all the reason to believe that it was straightforward.

    What other advice do I have?

    You will save a lot of time thanks to this solutions ability to run parallel testing. My team saves about a week and a half on our deployment schedule. Just be sure that you have settled on your design prior to implementation.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Tech Lead at a pharma/biotech company with 10,001+ employees
    Real User
    It reduces the testing time and covers most scenarios
    Pros and Cons
    • "Sauce Labs provides us with more combinations to test, so we can keep adding platforms and devices to our network. That's been a very seamless experience. Let's say there's an iOS or a private device we need. Sauce Labs has helped get all those set up when needed."
    • "I may not know what should be improved on the platform, but I think it could offer a greater variety of testing solutions. I know there might be competing solutions that cover more things I'm unaware of, and it could expand a bit more. We've done a lot of automated testing scenarios, and that's all the rage these days, so I think Sauce Labs could look at potential ways to improve and cover other scenarios."

    What is our primary use case?

    Our company has different divisions, like research and development, manufacturing, and research laboratories. There's also a department called Global Infrastructure Services. In total, we have five or six areas. In some cases, Sauce Labs is only used in a segment of a department. Take, for example, research and development. They might not use it in the laboratories, but they have a digital health section that works on apps or digital solutions for medical diagnostics.

    In this example, the primary use case is developing different software applications for medical diagnostics. They use Sauce Labs for testing applications in different operating systems and environments. We do regression testing across various ways someone might use an application. Instead of securing physical machines, they use Sauce Labs to do that. Many of the applications are internal to the company, so we use Sauce Connect to link up to the internal network to test those applications.

    We do a combination of parallel and sequential testing in Sauce Labs. Whenever you log in, you see all the tests on the dashboard. You have concurrent and parallel tests running, but some people will also use a sequential one.

    How has it helped my organization?

    Before we started using Sauce Labs, they needed to use physical machines to test applications. We required a Mac or Windows laptop to try different combinations, so it used to be a cumbersome process. Sauce Labs made testing more straightforward, so we could run more elaborate tests. 

    Sauce Labs provides us with more combinations to test, so we can keep adding platforms and devices to our network. That's been a very seamless experience. If there's an iOS or a private device we need, Sauce Labs has helped get all those set up when needed.

    As platform administrators, we get requests from different departments. Project teams using Sauce Labs come to us and say, "We have these three or four devices, but we might need another one that is private to us with this combination of operating systems and browsers." We get those spun up in Sauce Labs and have customer support. 

    The ability to run tests in parallel has cut down our test execution time for every application we use. The more we can reduce testing time, the easier it is to deploy an application, so running parallel tests is obviously an excellent way to do that.

    What is most valuable?

    It offers a decent range of browser-OS combination simulators and mobile device emulators. Our company tries out lots of different applications and solutions. It's not just for product development but for any application we use internally. For example, we use JIRA for project management, but we tried many solutions before deciding on that particular one. 

    We've been using Sauce Labs for a long time, even though there were other candidates we could have used. It's a robust platform that has met our needs, especially with Sauce Connect. We have internal applications that aren't available externally, and Sauce Connect makes that possible.

    What needs improvement?

    Nothing comes to mind right now, but this might be because I'm an administrator and not an end-user who is familiar with the application's use case. If we have teams asking for something that isn't already there, we communicate that back to Sauce Labs. That hasn't happened so far in the time that I've been in this role. 

    I may not know what should be improved on the platform, but I think it could offer a greater variety of testing solutions. I know there might be competing solutions that cover more aspects that I'm unaware of, and it could expand a bit more. We've done a lot of automated testing scenarios, and that's all the rage these days, so I think Sauce Labs could look at potential ways to improve and cover other scenarios.

    For how long have I used the solution?

    I work for the IT team of a pharmaceutical company, but different departments use Sauce Labs. I belong to the team that manages all applications the company uses, so we provide Sauce Labs for other users but don't use the product ourselves. I support it from the administrative side, but I've only been in this role for a few months. 

    What do I think about the stability of the solution?

    I haven't seen any downtime in the few months that I've been in this role, and we haven't heard any complaints about latency, but we've always used a cloud-based application. We wouldn't know the difference. We could compare if we had an on-premise version, but we've always used the cloud version, and it's the fast version.

    What do I think about the scalability of the solution?

    We have purchased a consistent amount of licenses over the years, and we are looking to grow with a focus on automated testing this year.

    How are customer service and support?

    Sauce Labs' support has been awesome. We have monthly support calls to catch up on any requests that we might have. When there is a new update or release, somebody from Sauce Labs comes to talk to us. It's not just us, but all users. It's like an open house. We have been planning that for some time now, but COVID has put a damper on those plans. When we meet monthly, we share any findings on our site, and they come up with best practices to share with us as well. We are satisfied with Sauce Labs' customer service.

    What's my experience with pricing, setup cost, and licensing?

    A supply management team deals with pricing and licensing. We look at the utilization of licenses every month and see if we need to purchase new ones then shoot them an email saying we need more. We don't get into the inner workings of licensing and procurement.

    What other advice do I have?

    I rate Sauce Labs eight out of 10. There are more and more different testing scenarios for various machines and environments we use every day. Sauce Labs provides a robust platform to test combinations of what a user might encounter when accessing any application.

    Sauce Labs reduces the testing time and covers all the possible scenarios that could come up, so we don't have to worry about any scenario that we might not have missed testing manually. It's a great platform to cover all your bases.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    Sr Staff Software Engineer, QA Enablement at a tech vendor with 501-1,000 employees
    Real User
    Easy to set up, saves us time, good parallelization capabilities
    Pros and Cons
    • "The most valuable feature is cross-browser, cross-OS, cross-mobile device testing."
    • "We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available."

    What is our primary use case?

    We have several healthcare products across the healthcare continuum, and we use the Sauce Labs platform to test our applications across different browser and OS combinations. We also use it to do mobile testing across different mobile devices that we may not have. It's not easy to set up these different configurations, so the cross-browser and cross-OS platform in the cloud gives us the ability to test across these different configurations without having to set them up or maintain them.

    At this point, we are trying to focus on API testing.

    My role is with a central team that helps other teams. If another team is struggling then we reach out to them and offer assistance. Because of this, I am familiar with how some teams are using the product. If on the other hand, a team is doing well and doesn't reach out, then I don't have any insight into how they are using the solution or how well things are going. 

    How has it helped my organization?

    We've had an event where I worked with the vendor to coordinate an automation day at my company. They worked with our QA engineers through a training process where we had a test repo that we were able to run across different browser and OS combinations using the platform.

    Technical support is very important because it helps us get through issues faster. When we are trying to integrate with a new tool, as is the case now because we're trying to move to GCP, we have a dedicated solution engineer to work with us. The engineer is helping us as we ramp up with GCP, getting our automation to work.

    We went through this a few years ago with Azure. Learning a new tool is challenging enough, and then trying to make it work with our existing processes is even more difficult. For example, we need to enable our test automation to run on the new platform and have everything work together. It can be challenging, not to mention that mobile testing is still evolving. This is why support is important. They have helped us with our questions and to get through any blockers.

    For the most part, the product is optimized for automation and integration with the major CI/CD platforms and developer tools. That said, our constant complaint is that it is kind of slow. We are trying to figure out ways that we can work around the slowness, and what things we can do better.

    We have a lot of products here in the company, and not everyone is leveraging the cross-browser testing platform that is available at Sauce Labs, but we have more and more teams adopting it in the last three years. We are trying to collect more insights into how our users are accessing our applications and using that insight to cover all those different testing configurations using the Sauce Labs platform, so we are trying to leverage the platform more in our internal testing.

    For now, Sauce labs covers our needs with respect to the number of browser/OS combinations and mobile devices. However, having more availability and looking into different strategies for making things faster are important. Our goal is always to get faster feedback on a build, which means if it takes a long time to run an automation suite then it's less likely that we'll use it. We're always seeking ways to add capabilities and expedite things.

    Having the ability to test our application on additional browser/OS/mobile device combinations opens up additional revenue streams for us because we have a larger user base. It gives us the confidence that the application we are putting out will generate a positive user experience because we've covered all of those testing combinations.

    We've gone through a lot of OKRs and have had a general goal to reduce the time it takes for us to test applications. This product enables us to run tests in parallel, which helps to make things faster. I don't have any concrete or documented examples of how long it would take to complete our testing before using the tool because we've evolved the way we write automation.

    There were times where it would take half a day to run an entire suite of tests, and through some of the improvements we've made, with and without the tool, our goal has been to be able to run a smoke test in 30 minutes. Some teams are able to achieve that, whereas others cannot.

    There is really no one answer that fits because we have teams that are across the spectrum. Some are working with mature products and some are just getting started. I know of one of our teams that optimized their automation and with Sauce Labs, they have been able to take their automated test time down from three hours to approximately 30 minutes. This is only one team out of many but it demonstrates a significant change.

    What is most valuable?

    The most valuable feature is cross-browser, cross-OS, cross-mobile device testing.

    The parallelization of automated tests is a best practice that I recommend. Some of the teams are able to do that, although some cannot because of the way the tests are formatted. If their configuration allows for it then it is definitely something that I recommend they take advantage of.

    Running testing in parallel has worked well for some teams. In one instance, a team was able to split their tests so that they ran across two platforms. Running these two configurations, they initially cut the three-hour testing time in half. Within each of those configurations, we were able to parallelize even further within our test framework, and ultimately, we were able to get the test time down to 33 minutes.

    What needs improvement?

    The process can be a little slow to configure and get started. It's a cloud platform and you're trying to run automated tests. It first has to be configured, then the test downloaded, then you have to configure the environment that you're going to run on, then kick off the test based on what your needs are. All of this takes time.

    We have faced challenges with the availability of mobile devices. There was once or twice where there were no mobile devices available. These instances may have been due to a service outage.

    For how long have I used the solution?

    I have been using Sauce Labs for approximately five years.

    What do I think about the stability of the solution?

    We run our testing on a daily basis and I don't think we've run into any stability issues. Nothing that I am aware of has been a concern.

    How are customer service and support?

    As we run into issues with anything that we are trying to do or integrating with to run our test automation, we reach out to their support team. Anytime that we need something and we need a quick answer, my first recommendation to anyone that reaches out to us for help is to call support and get it logged. This way, we can get someone to start responding.

    Some of the support team members are better than others. Overall, I would rate the support between six and seven out of ten.

    Which solution did I use previously and why did I switch?

    We did not use another similar solution prior to Sauce Labs.

    Also, personally, I have not used any other cross-browser testing platforms.

    How was the initial setup?

    I was not involved in the initial setup or getting it integrated with our automation test framework, but I think it was fairly easy.

    What about the implementation team?

    The initial setup was done in-house.

    As a SaaS product, it is always up to date.

    What's my experience with pricing, setup cost, and licensing?

    The pricing is definitely on the higher end, and there are other options that are more cost-effective. One of our teams that was using Sauce Labs separately decided to go with a different solution that was less costly.

    Which other solutions did I evaluate?

    Prior to choosing Sauce Labs, the company evaluated BrowserStack. I was not part of the decision process but the differentiating factor may have been the support.

    What other advice do I have?

    We have our internal framework and we basically use Sauce Labs for cross-browser capabilities and mobile device capabilities. Since we have our own framework, we have not explored any functional testing capabilities that the platform has.

    The vendor has acquired API Fortress, and it is available within our account for free to try out. That is something we're going to have a demo next week to look into because, at this point, we want to start focusing on API testing. It's helpful that we are able to run our UI tests across different platforms but we would like to focus more on the other layers of the application. We would like to test under the hood, the business layer, more.

    I would rate this solution an eight out of ten.

    Which deployment model are you using for this solution?

    Public Cloud
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor.
    PeerSpot user
    LMTS, Software Engineering at Salesforce
    Real User
    The ability to test the visual aspects of our apps' UI has been crucial
    Pros and Cons
    • "Sauce Labs helps us identify the root causes of bugs. The solution offers a lot of flexibility by providing the latest iOS and Android emulator versions, and even the Appium updates when it comes out in the market."
    • "Sauce Labs' dashboard could be improved by adding more filters and allowing more customization options. There was one instance where the dashboard on the Sauce Labs UI didn't meet our requirements, so we had to use the Sauce Labs API to create some apps and dashboards on our own. The API endpoints could be a little more robust and customizable."

    What is our primary use case?

    I'm on a team called the "Quadrant Three Mobile" team. Our focus is mobile testing and mobile browser testing. Sauce Labs provides iOS simulators and Android emulators that we use in our tests. It's running on our private cloud, and we have a Sauce tunnel set up to communicate between our environment and the Sauce Labs environment.

    We don't often use the Sauce Labs UI to track the tests we need to do. We have a UI on our end to do that. However, we log in to the web interface to view the dashboards to see the test strength and what was triggered or to search for specific tests exact testing. The Sauce Labs dashboard has links to videos of the actual test runs, which helps us deal with debugging issues and test failures. 

    Quality control engineers are the main users of Sauce Labs at the company, but developers also use it for making the CRM code base. They run tests on Sauce Labs mid-development to ensure their piece of work is going well.

    How has it helped my organization?

    Sauce Labs helps us identify the root causes of bugs. The solution offers a lot of flexibility by providing the latest iOS and Android emulator versions, and even the Appium updates when it comes out in the market. 

    Those have been helpful because the apps we test on Sauce Labs are used by customers across the globe, and we want to be sure we're testing on the latest iOS and Android versions as well as older versions that our customers still use heavily. Sauce Labs has done a decent job of giving us all the versions we need. 

    What is most valuable?

    Sauce Labs' dashboards are a good supplement for our dashboards and UI. The web interface is useful when we want to triage a specific test run or overall across different jobs. It's also handy when we want to look at what results we get when we run a test across a whole account.

    Sauce Labs' ability to test the visual aspects of our apps' UI has been crucial. At the end of the day, mobile testing is not only about functionality but also making sure your UI looks good in different languages and works well. Sauce Labs supports screenshots whenever test failures happen, giving us a good idea if something is broken on the visual side.

    What needs improvement?

    Sauce Labs' logging features could be improved. For example, when we run a mobile test on iOS, we want a way to capture the logs generated. It has some drawbacks there. Sauce Labs isn't able to give us everything we test for. 

    Sauce Labs' dashboard could be improved by adding more filters and allowing more customization options. There was one instance where the dashboard on the Sauce Labs UI didn't meet our requirements, so we had to use the Sauce Labs API to create some apps and dashboards on our own. The API endpoints could be a little more robust and customizable. 

    For how long have I used the solution?

    We have been using Sauce Labs for four to five years now.

    What do I think about the stability of the solution?

    Sauce Labs' reliability is one of the most important aspects of the platform. They have a status page to notify if there is any downtime. We rely on Sauce Labs because we have a continuous development process here, and we are testing code that comes out through Sauce Labs' emulators. 

    However, we've had occasional performance issues. For example, there are times when devices need to be restarted or the UI takes longer to load and things like that. The virtual emulators and simulators have been good, and their hardware and infrastructure have been standing up well.

    What do I think about the scalability of the solution?

    I think Sauce Labs has proven that their platform is pretty scalable because they give us a broad selection of versions available for iOS and Android. They also support their emulators with good virtual machines, so Sauce Labs can scale their infrastructure to their customers' needs quite well.

    I don't know exactly how many users are currently working with Sauce Labs in my company, but it's in the hundreds. We'll certainly expand usage in the future because we're planning to offer new kinds of mobile apps, and when those come out, we'll need to test them using Sauce Labs. As the number of products we offer increases, the usage will too. We use it for nearly all of our functional testing.

    How are customer service and support?

    I rate Sauce Labs' support at about seven out of 10. We put in certain requests with Sauce Labs, and sometimes it takes a while to get solutions implemented, depending on the priority.

    How was the initial setup?

    My team didn't deploy Sauce Labs in our environment. A different team handled that. We did the initial installation back in 2016, and I don't recall it being very complex. 

    They had some binaries and decent instructions on how to set up Sauce Labs' tunnels. They also give instructions on how to capture logs of those tunnels to see traffic between your environment and the Sauce environment. All of that is pretty well documented, and support teams have been quite helpful. Whenever we needed something out of the box, we've been in touch with them for it. 

    What was our ROI?

    I can't really quantify a return on investment, but testing on Sauce Labs has helped us catch issues and fix problems much sooner than we would have without it. That has saved us money.

    What other advice do I have?

    I rate Sauce Labs eight out of 10. I recommend it for mobile testing. Sauce Labs is a reliable vendor, and you can trust their network. They have a good platform, with broad support for various mobile OS versions. Their availability is also high in the region, so we seldom see outages. The dashboards on their website for tracking your mobile testing strength are helpful. Overall, it's a good product for anyone wanting to do mobile testing. 

    The main aspect I've learned from working with Sauce Labs is that it's not easy to maintain a range of devices or virtual environments and provide it for mobile customers. We have tried to do that at my company, and it's challenging. I definitely like what Sauce Labs is doing in terms of providing that reliable network for us. We're also learning more about our own product through testing on Sauce Labs. The sooner you test, the quicker you identify issues in your own application. 

    Which deployment model are you using for this solution?

    Private Cloud

    If public cloud, private cloud, or hybrid cloud, which cloud provider do you use?

    Amazon Web Services (AWS)
    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor. The reviewer's company has a business relationship with this vendor other than being a customer: Partner
    PeerSpot user
    Head of Automation R&D at Applause
    Real User
    Enables us to support multiple versions of browsers, devices and OS combinations, but OTT support would make it more complete
    Pros and Cons
    • "They update for the latest browsers and mobile phones and support a lot of combinations. They have 1,000-plus desktop combinations and browser versions, which is really great. We need that at Applause. The all-in-one testing suite aspect of it is really important because most of our clients prefer to go to one place."
    • "Latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test cases."

    What is our primary use case?

    At our company, Applause, we offer software testing as a service and we always get a lot of interesting, uncommon or challenging use cases from our clients. We sometimes get ones that require specific devices or browsers to work. For example, we have clients who want to mix testing on desktop browsers and apps or test on multiple apps to achieve some kind of scenario; perhaps you are at a restaurant, and you are ordering something on your personal phone/tablet, which shows up on the restaurant's tablet or desktop browser. 

    Our clients are not only looking for executing the test cases manually, but their target is to automate all of them and be able to integrate that into their CI/CD pipeline and get faster feedback about the stability of the changes that the development team produces on a daily basis.

    Sauce Labs covers all of our automation needs and also allow us to do manual testing in case we are verifying bugs or testing something else.

    How has it helped my organization?

    My main focus is on automation testing and Sauce Labs is an integral part in our success as a company as they offer 1000+ different combinations of desktop browsers, real devices & OS versions. This allows us to pretty much cover all the automation test scenarios we need. We've also integrated them in our internal SDKs and are using their platform on a daily-basis to test the quality of our clients' apps and websites.

    The number of the real devices they offer is also very important to our business, because most clients want to support multiple devices & OS versions. We have clients who want us to run automation tests on Android 7, 8, 9, 10, 11 and sometimes even on Beta versions. Sauce Labs offers all of that, enabling us to support all of these versions, which is really nice. The same goes for their desktop browsers, although most of our clients usually use either the latest version or the one before that. I haven't seen a client who wants to use and old browser version like Firefox 50, but it's a great feature that Sauce Labs offers, in case it is needed for any reason.

    In terms of delivering software faster by getting feedback with every commit, it depends on the client and the system under test (SUT); some of our clients are testing manually and that would take them days, others have a CI/CD pipeline and run some smoke tests after each commit. Part of our job at Applause is to enhance the overall testing process of our clients & automate it as much as possible. Instead of taking days to do a full regression, it might take a couple of hours to get the same results. 

    Because we're working with Sauce Labs, and we're doing automation for our clients, they get a pretty fast cycle of feedback that allows them to make the right decisions at the right time.

    Without Sauce Labs, it would be really hard to achieve the software quality standards or the automation coverage that our clients need.

    What is most valuable?

    Towards the beginning of this year, we had to constantly switch between Sauce Labs and their legacy solution for real device testing, TestObject. This was causing a lot of issues or confusion for our engineers & clients. Now, with the new unified platform, we get everything we need in one single place and it is so intuitive and straightforward. You can easily do manual testing, view running automation scripts or switch between data centers from one single place.

    Overall, we're very happy with the combinations of browsers and devices they offer, especially because they always have multiple instances of the same device. If you want an iPhone 12 for your tests, you can get five of them for example and run your test scripts in parallel, which reduces the total execution time.

    Sauce Labs also allows specifying the browser version as "latest", which means that they automatically pick the latest version they support and run the tests against it. Previously, we had to manually change the browser version in our scripts whenever a new one becomes available and it was wasting a lot of time (as it needed to be done for a lot of projects).

    Their platform also offers some cool features if you're working with apps. You can just pass them a URL to an app build that is stored somewhere on your servers or on AWS, and they will install that on the phone/tablet and run tests against it on the spot. This has allowed for an easier integration with our SDK, as opposed to competitors who don't offer such feature.

    What needs improvement?

    The market is changing and we're seeing a lot of shift towards testing on OTT devices, like Apple TV, Fire TV Sticks, Chromecast and Roku. This is really an emerging market and has the potential to grow exponentially. We've been getting a lot of clients asking us to test on OTT devices, either manually or via automation. We have been able to automate testing on OTT devices, but it is done with local devices hosted on our premises, so we can't scale that fast or cover all of the huge demand we are seeing. We're looking for the ability to do this in the cloud, so if Sauce Labs offered such a feature, that would be really great. Working on OTT devices usually includes testing on desktop browsers or phones/tablets and having all of those supported in one platform would be ideal.

    While Sauce Labs pretty much covers everything we need, another exception is HarmonyOS. I know it's a new thing and it's not supported by a lot of providers, but it is something that clients have started asking for. That is something we'd be really interested in seeing, for both manual and automated testing.

    Also, latency, due to Sauce Labs being a cloud-based solution, has been a concern. We work in different continents and countries, but last time I checked, Sauce Labs was only offering two data centers, one in the EU and another in the US. If you're not in either of those two places, you would have latency and issues running your test scripts. However, I'm working with clients in different countries and we've seen some latency issues, depending on the country, although it's not huge. In comparison to other providers who have data centers in different places, there is some noticeable latency.

    For how long have I used the solution?

    I've been using Sauce Labs for over five years.

    What do I think about the stability of the solution?

    Sauce Labs is pretty stable in comparison to other providers. We still get some issues every now and then or random failures, especially when there is a new OS or browser version, but it's not a big deal and we can easily contact Sauce Labs' support to get things fixed.

    What do I think about the scalability of the solution?

    One of Sauce Labs' competitors does not force you to select the data center you want to run against. They just figure out which data center is closest to where you are executing your test scripts from. They offer a single, unified endpoint or URLs when it comes to automation and handle everything else internally. When we're writing automation scripts, we don't need to specify, "I want to run on the European data center," or "I want to run on the American data center." We can just say, "I want to run a test case," and depending on the location you come from, the platform is smart enough to direct it to the nearest data center to reduce latency.

    This may not be directly related to scalability, but this kind of capability would make it easier for us to build our SDKs faster and focus on other features, which in return would allow us to scale faster as a company. The learning curve for newcomers would also be easier, because they wouldn't need to worry about figuring out which data center to run against.

    While I love the fact that we can specify the data center ourselves, because it gives us more freedom, I would love to see more data centers in different places around the world, to reduce latency, and the selection of the data center implicitly done by the platform, so we don't need to worry about it.

    How are customer service and support?

    We open support tickets, and they're pretty responsive, they get back to us on time. They're really good about fixing things and making their platform more stable.

    I've run into a couple of situations where tickets have been left for weeks without feedback, but that has not happened often. When it does happen, I can go to our customer success manager and say, "We need some attention to these tickets." She can take care of it and that makes the support pretty good.

    How would you rate customer service and support?

    Positive

    What other advice do I have?

    Pay attention to selecting the correct data center, otherwise you would run into some latency-related issues.

    Although we are not using visual regression testing, that is something that would be really interesting, because a lot of our media clients want to test visual validations on their apps. If Sauce Labs is offering that out-of-the-box, it would be really interesting for us.

    Disclosure: PeerSpot contacted the reviewer to collect the review and to validate authenticity. The reviewer was referred by the vendor, but the review is not subject to editing or approval by the vendor. The reviewer's company has a business relationship with this vendor other than being a customer: Partner
    PeerSpot user
    Buyer's Guide
    Download our free Sauce Labs Report and get advice and tips from experienced pros sharing their opinions.
    Updated: April 2024
    Buyer's Guide
    Download our free Sauce Labs Report and get advice and tips from experienced pros sharing their opinions.