Android UI Tests on Every PR: emulator.wtf and Marathon Cloud — Part V

Evgenii Matsiuk ·

This is the last article in my series focused on finding efficient solutions for running UI tests on each PR. I will explore the two most promising and attractive options: emulator.wtf and Marathon Cloud. Finally, I will draw conclusions based on the findings of this series.

Marathon Cloud

Marathon Cloud is built on the widely-used open-source UI test runner, Marathon Runner. All the functionalities offered by Marathon Runner are automatically supported by Marathon Cloud.

Marathon Cloud specializes in mobile testing for Android and iOS using only native tools. At the moment, Appium is not included in Marathon’s services.

Only Marathon Cloud provides a “15-minute” promise about the time execution of any number of tests (50, 1000, 4000, doesn’t matter) and frees a user of necessity to predict the appropriate number of parallels to fit in some particular time frame. That’s why the number of parallels may vary from one run to another.

To run UI tests successfully, managing flaky tests is essential. Only Marathon Cloud offers these features seamlessly, without requiring developers to put in extra effort. Also, Marathon Cloud offers the same functionalities for iOS at the same price as Android — you can use a single solution with consistent settings for both platforms at the same time.

Series Conclusion

In this series, I considered: Marathon Cloud, Firebase Test Lab, BrowserStack, emulator.wtf, SauceLabs, AWS Device Farm, Perfecto Mobile, LambdaTest.

For PR runs, opting for the “Pay for spent hours/minutes” policy tends to be the most cost-effective choice. This policy is offered by only 5 solutions: Marathon Cloud, emulator.wtf, SauceLabs, Firebase Test Lab, and AWS Device Farm.

AWS Device Farm falls short due to its lack of emulators and parallelization, as well as its high cost of $10.2 per hour. SauceLabs is decent but not the best in terms of time and price at $4.8 per hour.

As a result, only three options remain competitive: Marathon Cloud, emulator.wtf, and Firebase Test Lab.