Skip to content
←Testing

// Testing · 09

Mutation Testing

Source codeleads tosurvivorsMutantsleads toTests kill or pass

Break the code on purpose; good tests must notice every break.

How it fits together

Mutation Testing structureSource codeif (total > 100)return discount;Mutant A> becomes >=Mutant Breturn 0 insteadTest runfull suite per mutantKilleda test failedsuite noticedSurvivedall tests greengap to fill1 mutate1 mutate2 run tests2 run tests3 fails3 passes
createsuses / callsfails / error path· numbered steps replay in order
  • Source code creates Mutant A (1 mutate)
  • Source code creates Mutant B (1 mutate)
  • Mutant A uses / calls Test run (2 run tests)
  • Mutant B uses / calls Test run (2 run tests)
  • Test run uses / calls Killed (3 fails)
  • Test run fails / error path Survived (3 passes)

Spot it

  • Coverage is 90% but bugs still ship
  • Tests still pass after an operator is flipped
  • Unsure a test would catch a regression

It costs

Slow, since each mutant reruns tests; it needs a scope and a timeout to stay practical.

Go deeperThe full explanation and what to ask for · 1 min readExpand

A mutation tool makes small legal changes, such as > to >=, a return value replaced by null, or a statement removed, and runs the tests for each one. A mutant that makes a test fail is killed: the suite noticed. A mutant that leaves everything green has survived: nothing checked that behaviour. The mutation score is killed mutants divided by all mutants, less those that cannot compile or are equivalent.

Coverage says a line ran; mutation testing says a result was checked. A test with no assertions runs the code and passes, so coverage stays high while every mutant survives. Survivors point at the exact boundary or condition the suite leaves unpinned, which makes them a better to-do list than a percentage. Equivalent mutants change nothing observable and cannot be killed, so review them rather than writing a test for them.

Tools exist for most ecosystems (Stryker, PIT, mutmut). Each mutant reruns tests, so run on changed code in CI or nightly rather than on every commit of a large solution. When NOT to: code that is mostly framework glue or has little logic yields noisy mutants, and a score there tells you little.

Ask for it

  • mutant
  • killed mutant
  • survived mutant
  • mutation score
  • equivalent mutant
  • incremental run

Run mutation testing on the Billing project, changed files only, against its unit tests. List every surviving mutant with file and line, grouped by operator. For each survivor, add a test that would fail under that mutation. Target a mutation score of 80% or more on the changed code.

Connects to