Skip to content
←Testing

// Testing · 10

Flaky Tests

Same code, two resultsleads totime, orderHidden inputleads toFind the input

Passes and fails on the same code: a hidden input is leaking in.

How it fits together

Flaky Tests structure«client»Checkout testsame code, same inputfails 1 run in 20«external»System clockreads now()can cross midnightShared cachestatic staterows left by othersRandom failuredepends on the houror on test order1 reads now1 reads cache2 late hour2 leftover row
uses / callsfails / error path· numbered steps replay in order
  • Checkout test uses / calls System clock (1 reads now)
  • Checkout test uses / calls Shared cache (1 reads cache)
  • System clock fails / error path Random failure (2 late hour)
  • Shared cache fails / error path Random failure (2 leftover row)

Spot it

  • Re-running the pipeline turns it green
  • Fails only on CI agents, never on laptops
  • Someone added a sleep to make it pass

It costs

Fixing takes refactoring time; quarantine keeps the build green but hides the bug.

In code

using Microsoft.Extensions.Time.Testing;using Shouldly;​public sealed class ExpiryPolicy(TimeProvider clock){    public bool IsExpired(DateTimeOffset issuedAt) =>        clock.GetUtcNow() - issuedAt > TimeSpan.FromMinutes(30);}​public sealed class ExpiryPolicyTests{    [Fact]    public void IsExpired_After31Minutes_ReturnsTrue()    {        // Arrange        var start = new DateTimeOffset(2026, 1, 1, 23, 50, 0, TimeSpan.Zero);        var clock = new FakeTimeProvider(start);        var policy = new ExpiryPolicy(clock);        clock.Advance(TimeSpan.FromMinutes(31));​        // Act        var expired = policy.IsExpired(start);​        // Assert        expired.ShouldBeTrue();    }}
Go deeperThe full explanation and what to ask for · 1 min readExpand

A flaky test has a hidden input. The usual suspects are the wall clock (it passes before midnight and fails after), test order (one test leaves a cache or a file behind), shared state between parallel runs (two tests write the same row), unseeded randomness, real network calls, and fixed sleeps that are enough on a laptop and not on a loaded agent. Find the input by making it explicit and repeatable, not by rerunning until the test passes.

Replace sleeps with waits on a condition with a timeout, such as poll until the row exists for up to five seconds, and fail with a clear message. Inject the clock and the random source so the test controls them: advancing a fake clock by 31 minutes is deterministic, waiting 31 real minutes is not. Give each test its own data and never depend on order.

Blind retries hide the signal: a test that passes on the third attempt says nothing about the second. Quarantine is a short, tracked holding state with an owner and a deadline, not a parking place. When NOT to: a test of real timing, such as a rate limiter, must use real time, but bounded and measured, not a sleep tuned by hand.

Ask for it

  • flaky test
  • quarantine
  • order dependence
  • injected clock
  • deterministic
  • test isolation

Find why the expiry checkout test fails about 1 run in 20: rerun the suite 50 times with the test order shuffled, log the clock value and any shared static state, then inject the clock so the test sets it. Quarantine only with a linked ticket and a two-week deadline.

Connects to