Testing Fundamentals: Reliability

Working with any codebase requires verifying whether the result of a change or even the current state is valid. To do that, we interact with our software via direct software interface such as calling methods or interacting with the customer surface via calling CLI or interacting with GUI. Sometimes we even automate such actions via other code that we call automated tests.

Unfortunately, when executing such code, we often face unexpected behaviour. A perfect system doesn’t have any unexpected behaviour, however this is almost impossible to achieve at scale. If the amount of unexpected behaviour is under a certain threshold then the system is considered reliable. Let’s understand why unexpected behaviour exists and what we can do about it to make our system more reliable.

Determinism

A simple example of undesirable determinism: a UI that switches between a dark theme at night and a light theme during the day. Tests that depend on the visual state will pass or fail depending on the time of day they run. This is deterministic behavior, but it’s undesirable for testing.

Flakiness

Reliability of testing code depends fully on the determinism. Code that is not deterministic is called flaky code. Causes of flakiness might be both under your control and outside of your control.

To solve the flakiness is to have control over the behaviour and logic of your code including all inputs and the implementation. If such fix is not an option then a pragmatic approach with retries helps mitigate flakiness by sacrificing the cost of execution.

Retry Strategies

Implementing retry logic can be done in the testing framework, but it’s much better to implement it once for every technical stack and reuse it via a general-purpose test runner.

Marathon implements all of the above retries and their cost centers via configuring flakiness strategy, retry strategy and the uncompleted tests quota. Most of the solutions you can find currently only implement the post-factum retries which result in sub-optimal test execution time and higher cost.