Mutation Testing — Coverage vs Mutation Score.

⚖️Two metrics that sound the same

Line coverage answers: "did my tests RUN this line?"

Mutation score answers: "would my tests NOTICE if this line were wrong?"

Coverage = execution. Mutation = detection. A test file with zero assertions gets 100% coverage.

🧪The 60-second demo

def is_eligible(age, score):
    return age >= 18 and score > 700

def test_eligible():
    assert is_eligible(25, 800) is True
    assert is_eligible(10, 500) is False

Coverage: 100%. Green build. Ship it?

Now a mutation tool rewrites the source, one tiny change at a time, and re-runs your suite. Each variant is a "mutant". If the suite still passes, the mutant SURVIVED — that's a bug your tests can't see.

age >= 18  →  age > 18     survived (nobody tested age exactly 18)
score > 700 → score >= 700 survived (nobody tested exactly 700)
and → or                   survived (10>=18 False, 500>700 False → still False)

3 survivors out of ~6 mutants → mutation score ~50%, from a file showing 100% coverage. And all 3 are the exact off-by-one/boolean bugs that reach production.

🛠️Running it

Python:  pip install mutmut && mutmut run --paths-to-mutate src/billing.py
JS/TS:   npx stryker run
Java:    PIT (pitest)

Read the output as a to-do list of missing test cases, not as a grade.

🪤What practitioners get wrong

1️⃣ Running it on everything. Cost is roughly (number of mutants) × (suite runtime). A 3-minute suite and 400 mutants = 20 CPU-hours. Scope it: changed files in PRs, or nightly on your 5 most dangerous modules (pricing, auth, permissions). Never a blanket repo-wide gate.

2️⃣ Chasing 100%. Equivalent mutants are semantically identical to the original (swapping < for != when the bound is enforced elsewhere) and are literally unkillable. 100% is a trap; 70-80% on money/auth code is a strong target. Leave DTOs, config and glue alone.

3️⃣ Killing mutants the lazy way. Asserting on internals or snapshotting the whole object kills the mutant and leaves you with a brittle test. The right fix is almost always a new input at the boundary: is_eligible(18, 701), is_eligible(17, 5000), is_eligible(25, 700).

4️⃣ Confusing a survivor with dead code. A surviving mutant means untested behavior. If NO test exercises the line at all, that's a coverage gap — fix that first, then re-run mutation.

📊Quick side-by-side

Coverage   — cheap, fast, gameable, good floor
Mutation   — expensive, slow, honest, good ceiling

Use coverage to find code nobody touched. Use mutation to find code nobody verified.

🎯Your exercise

Pick the one pure function in your codebase you'd least like to be wrong (discount calc, permission check, retry policy). Run mutmut/stryker on just that file.

Then report back two numbers: how many mutants survived, and of those, how many were real missing test cases vs. genuinely equivalent mutants. The ratio tells you more about your suite than any coverage badge.