Android UI Tests on Every PR: SauceLabs, AWS, LambdaTest, Perfecto — Part IV

Evgenii Matsiuk ·

Let’s continue our investigation of existing solutions for running Android UI tests on each PR. Today, I will have a look at SauceLabs, AWS Device Farm, LambdaTest, and Perfecto Mobile.

Disclaimer: I hold the position of Co-Founder at MarathonLabs. While I have attempted to remain impartial, I acknowledge that some of my views may be biased.

SauceLabs

SauceLabs is a great solution if you want to run tests for different products — mobile apps, desktop apps, and websites — in the same place. It offers a decent testing experience with real devices and emulators.

However, for the specific use case of running UI tests on every PR, SauceLabs may not be the most cost-effective at $4.8 per hour.

AWS Device Farm

AWS Device Farm is an app testing service for testing Android, iOS, and web apps on real physical phones and tablets hosted by AWS. It supports native and hybrid apps including PhoneGap, Titanium, Xamarin, Unity, and other frameworks.

AWS Device Farm lacks support for parallelization — limited to 5 devices by default with the option to request an increase. The cost is $10.2 per hour. It does not offer emulators, which impacts both stability and speed.

LambdaTest

LambdaTest did not offer a trial for their solution at the time of evaluation, making it difficult to assess for this comparison.

Perfecto Mobile

Similar to LambdaTest, Perfecto Mobile did not offer an option to trial their solution.

Conclusion

I have explored four alternative options for running UI tests on each pull request. Frankly speaking, only SauceLabs seems like a viable solution for this purpose. AWS Device Farm lacks support for parallelization, and both LambdaTest and Perfecto Mobile do not offer the option to trial their solutions.