Running the whole Espresso suite on every pull request

An Espresso suite tends to end up as a nightly job. It starts as a check on every PR, the suite grows, and at some point nobody wants to wait for it before merging. So the full run moves to nightly and the PR keeps a smoke subset. CI is faster, but now regressions show up the next morning, after the change is already merged. Eventually the trust in nightly erodes and the check is done weekly or before each release.

This is not unique and almost every Android project goes through this stage, usually around 200-300 UI tests. What if the answer to having faster releases and high quality tests is to keep the whole suite on the PR and add devices until the wait is short again? The more you run it - the more confidence you’ll have in it.

Splitting by test count

The obvious way to add devices is sharding: split the test classes into N equal lists and give each list its own emulator. It looks balanced in the workflow file, but the tests are not equal. A test that launches the app, creates data and goes through checkout costs a lot more than one that checks a single validation message. One shard gets the slow tests, the other devices finish early and sit idle, and the PR waits for the slowest shard anyway.

Marathon Cloud works from time instead of test count, i.e. how long each test took in previous runs. It estimates the duration of the whole suite and sizes the device pool to finish in about 15 minutes, so a suite with two hours of device time gets roughly eight devices. Batches are sized from each test’s history as well, so one slow batch doesn’t hold up the whole run.

15 minutes is a target. The first run has no history and starts from a conservative default, so I treat the first few runs as measurements and keep the CI job timeout at 30 minutes or more (upload time counts too). Long tests are the other thing that pushes a run past the target: no amount of devices makes a run shorter than its longest test. If a test takes minutes, I’d rather split it by using deeplinks instead of repeating the same login->onboard->navigate flow that is shared between many tests.

Running it on every pull request

On the CLI side it’s one command:

marathon-cloud run android \
  --application app.apk \
  --test-application test.apk

These are the same two APKs you already build for Espresso, the app and the instrumentation APK. The API key comes from the console, and the CLI reads it from MARATHON_CLOUD_API_KEY, so it stays out of the command and the repo. If you’ve never run anything on Marathon Cloud, the older Android walkthrough covers the dashboard and the first run.

For GitHub Actions there’s an official action. This is the complete job from the CI/CD docs:

name: ui-tests
on: [pull_request]

jobs:
  marathon:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: 17
      - run: ./gradlew :app:assembleDebug :app:assembleDebugAndroidTest

      - name: Run tests on Marathon Cloud
        uses: MarathonLabs/[email protected]
        with:
          version: 1.0.67
          apiKey: ${{ secrets.MARATHON_CLOUD_API_KEY }}
          platform: Android
          application: app/build/outputs/apk/debug/app-debug.apk
          testApplication: app/build/outputs/apk/androidTest/debug/app-debug-androidTest.apk
          osVersion: 15
          systemImage: google_apis
          output: marathon
          link: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
          branch: ${{ github.head_ref }}

      - name: Publish JUnit
        if: always()
        uses: mikepenz/action-junit-report@v5
        with:
          report_paths: marathon/tests/omni/marathon_junit_report.xml

The action wraps the same CLI. The test step exits non-zero when any test fails, so the check goes red by itself, and if: always() still publishes the JUnit report after that. output: marathon is where reports and artifacts get downloaded once the run is done. The same action runs XCUITest and Maestro too, see mobile UI testing for how this looks beyond Android.

Not every CLI flag has an action input, --concurrency-limit for example. For those I install the CLI with MarathonLabs/setup-marathon-cloud and call it from a run: step, with --output to get the same downloads:

marathon-cloud run android \
  --application app.apk \
  --test-application test.apk \
  --output marathon

The result shows up in the normal PR checks and the full report is one click away. Most of the time the JUnit failure message is enough. When it isn’t, the run also has an Allure report, Marathon’s HTML report with the timeline, and a video and a device log for every test.

When staging can’t keep up

Going from a few shards to whatever the scheduler picks also tests things outside the test code. Tests that share an account or a fixture start interfering with each other, and staging gets hit by more devices at once than it was sized for. Marathon has always assumed that every test is isolated and can run on any device next to any other test. Shared state breaks that assumption, and with more devices it breaks more often.

To patch this problem temporarily use --concurrency-limit which caps the pool for that run:

marathon-cloud run android \
  --application app.apk \
  --test-application test.apk \
  --concurrency-limit 4

It’s a band-aid. A suite with two hours of device time capped at 4 devices takes about 30 minutes instead of 15, and you’re back to tuning a number by hand. I’d use it to unblock the PR, then fix the test data, give staging some headroom and drop the flag so the pool gets sized from history again.

Retrying a single test

When a test fails, Marathon Cloud schedules that one test again, on a different device than the one it failed on. If it passes on any attempt it’s reported as flaky, and if it fails every attempt it’s failed. Compare that with the “Re-run failed jobs” button in GitHub, which reruns every test in the job because of one.

The scheduler also keeps a history of outcomes for every test. If a test has been unstable recently, it can get preventive retries: it runs more than once from the start, without waiting for the first failure. That costs a bit of device time, but the verdict comes back sooner, which matters when the whole run is supposed to fit into 15 minutes and also measures healing quicker.

Then there are tests that never produce a result at all, because the device crashed, the process got killed or the batch timed out. Those count as uncompleted rather than failed and get re-run under their own quota (--retry-quota-test-uncompleted), so infrastructure problems don’t eat the retries meant for real failures.

Flaky tests count as passing for the exit code, so a green run can still have a few. They’re listed separately in the report. I still treat a test that keeps coming back as flaky as a bug, I just don’t want every flake to cost a full pipeline rerun.

The outcome history behind preventive retries isn’t shown in the console yet, so right now you can’t see why a test got extra runs up front.