We write code faster than we can read it. With AI-assisted coding, a function, a module, sometimes a whole feature appears in seconds. Tests often arrive in the same breath, written by the same assistant. They are green. But green only says the tests pass. It does not say they would notice if the code were wrong.
Absence of evidence is not evidence of absence:
Program testing can be used to show the presence of bugs, but never to show their absence. Dijkstra
Reading every line ourselves no longer scales. So we need tools that look into a codebase for us, without a human watching each step. Linters and type checkers do part of that. Coverage tells us which lines run. Mutation testing goes one step further: it breaks the code on purpose, thousands of times, and asks the tests each time: did you notice?
The machine does the tireless part. What comes back is a short list of places where the tests stayed silent. Human time is then spent only where it is needed: deciding which of those are real bugs waiting to happen, and which are loose ends that do not matter.
The idea is old, from the 1970s, and every major language has tools for it: mutmut and Cosmic Ray in Python, PIT in Java, Stryker in JavaScript and C#. This post uses mutation.py, a small single-file tool I wrote, because it is short enough to read from top to bottom. By the end, you will know how such a tool works inside, and why it is worth adding to your own project.
Picture a market inspector in a souk in Fès. Every week she checks the merchantsâ scales with a known weight. But how does the city know the inspector herself is careful? One way: secretly file a scale so it reads 50 grams heavy, and see whether she catches it. If she does, good. If the faulty scale passes her inspection, the city has learned something important about the inspector, not about the scale.
Mutation testing does exactly that to your test suite. Your tests are
the inspector. The tool makes a small, deliberate change to your code, a
mutant: a < becomes <=, a
+ becomes -, a line is replaced by
pass. Then it runs your tests.
In mutation testing âkilledâ is good, because a mutant must be killed, that applies to the code being checked, it doesnât apply to x-men.
A killed mutant reveals no gap in the test suite.
A survivor is not always a bug. Sometimes the change makes no real difference (an equivalent mutant). That is why the last part of the tool is for a human, who reviews each survivor and decides.
mutation playEverything starts in main.
It reads the command line with my tiny hand-written parser,
cli_read, then picks a subcommand with Pythonâs
match statement:
keywords, standalone, extra = cli_read(sys.argv[1:])
...
match standalone:
case ["play", *files]:cli_read
sorts arguments into three piles: flags like --verbose,
plain words like play, and everything after
--, which is passed untouched to pytest. No argument
library, about twenty lines you can read in one sitting.
cli_read surely deserves its own post, thatâs for another
time.
The play command does the whole job in five steps:
Then replay brings the survivors to you, one by one, to
review.
We now know the big picture. From here we build it from the bottom
up: first the small helpers that everything else relies on (the
most-used ones first), then the pieces made from them, and finally
play and replay again, which by then you will
be able to read line by line. Each section shows only a few key lines;
every function name links to its full code on GitHub.
To change code safely, mutation.py never edits text. It edits a tree.
Python ships a module called ast (Abstract Syntax Tree)
that turns source code into nested objects: a function holds statements,
a statement holds expressions, an expression like a + b
holds a, the operator +, and b.
Change one object in the tree, turn the tree back into text, and you
have a mutant that is always valid Python.
Three tiny functions do all the tree work. Almost every one of the 32 mutation kinds in the file calls them, which is why they come first.
ast_walk
visits every node, parent before children:
def ast_walk(tree):
yield tree
for child in ast.iter_child_nodes(tree):
yield from ast_walk(child)If yield is new to you: it makes the function a
generator, which hands out one value at a time instead of
building a full list. yield from hands out everything a
nested generator produces. Because the order is always the same,
every node gets a stable number: its position in the walk.
That number is the trick behind copy_tree_at:
tree_copy = deepcopy(tree)
return tree_copy, list(ast_walk(tree_copy))[index]Each mutant needs its own copy of the tree, so the original stays
clean. But after deepcopy, the node we wanted to change is
a different object. How do we find it in the copy? We walk the
copy in the same order and take the node at the same position:
simple, and a nice lesson, when objects get copied, a position can be a
better address than the object itself.
get_parent_field_idx
answers the opposite question: who holds this node, under which field,
at which position in a list? You need that to replace a node,
for example to swap a whole statement for pass. It searches
the full tree each time, which is slow on paper, but the code stays
short and easy to trust.
Each kind of mutation is a small class with two methods:
predicate says âcan I change this node?â, and
mutate produces the changed copies. The file holds 32 of
them. You do not need to read them all: once you have read two, you can
read any.
How does the program know all 32 classes exist? Nobody writes a list
by hand. The classes use Mutation
as their metaclass, the class that builds classes. Python calls
its __init__ each time a new mutation class is defined, and
it registers one instance:
obj = cls()
type(cls).ALL.add(obj)So writing a new class is enough to add a new mutation. Metaclasses are an advanced corner of Python; here you only need the idea: âdefining the class is the registrationâ.
passStatementDrop
asks the bluntest question: if this line did nothing at all, would a
test fail? It copies the tree, finds the statementâs parent with the
helpers we just met, and puts a pass in its place:
replacement = ast.Pass(lineno=node_copy.lineno, col_offset=node_copy.col_offset)
getattr(parent, field)[idx] = replacementIf such a mutant survives, that line is probably dead code, or your tests never check its effect.
MutateOperator
turns + into each of its siblings (-,
*, //, and so on), and < into
<=, ==, >. Notice
mutate uses yield inside a loop: one node can
give many mutants, handed out one by one.
The other classes follow the same pattern. A few, to show the range of questions they ask:
ForceConditional:
make an if always true, then always false. Are both
branches tested?ZeroIteration:
loop over an empty list. Do you test the empty case?MutateReturn:
return None, 0, False or
"" instead. Does anyone check the result?MutateIterator:
walk a loop in reverse or shuffled order. Does order matter, and is it
tested?InjectException:
raise the error a line could raise, like KeyError
on d[k]. Is the error path tested? This is the largest
class, because it guesses from variable names whether x[k]
is a dict or a list.A mutation run can take hours, and you may stop it halfway. So every
result goes into a notebook on disk: a SQLite file named
.mutation.db. SQLite comes with Python, needs no server,
and stores a whole database in one file.
database_open
is the most-called function in the program: a dozen places open the
notebook, read or write one thing, and close it again. Short visits keep
the code simple, and they let many workers share the file safely.
The Database
class creates three tables:
config: settings of the last run, like the test command
and the random seed, so replay can repeat them
exactly.mutations: one row per mutant, with an id, the file
path, and the change written as a compressed diff.results: only the survivors, with an optional human
classification.One detail worth stealing for your own projects is how the schema grows:
try:
self._conn.execute("ALTER TABLE results ADD COLUMN classification INTEGER")
except sqlite3.OperationalError:
pass # column already existsThe classification column came later in the projectâs
life. Instead of a migration system, the code simply tries to add it,
and ignores the error if it is already there. Old notebooks keep
working.
Each mutantâs id comes from make_uid:
6 bytes of the current time in milliseconds, then 10 random bytes.
Sorting ids then also sorts mutants by creation time.
A changed tree lives only in memory. To store a mutant, and to show
it to a human later, the program turns it into a diff: the few
lines that differ, with - for the old line and
+ for the new one, the same format git diff
shows you.
The heart of this step is iter_deltas.
Its first lines hide a subtle choice:
tree = ast.parse(source)
canonical = ast.unparse(tree)ast.unparse turns the tree back into text, but in
Pythonâs own standard layout, without your comments or your spacing. The
program compares mutants against this canonical text, not
against your file. Why? Because both sides of the diff then come from
the same printer, so the diff shows only the real change, never noise
from formatting.
Then, for every node in the walk and every mutant it produces, two filters apply:
interesting
drops mutants on lines your tests never run. If no test runs a line, of
course no test can catch a change there; reporting it would only bury
the useful results.The survivors of both filters go through diff,
a thin wrapper over the standard difflib.
One level up, mutation_create
does this for a whole file and compresses each diff with
zlib before handing them back. Diffs are mostly repeated
text, so they shrink well, and less data travels between processes.
A stored diff is useful only if we can turn it back into code. That
is the job of patch.
It reads each hunk header, a line like
@@ -12,3 +12,3 @@ meaning âat line 12 of the old text, 3
lines changeâ, and copies the source, swapping in the +
lines. A comment above it admits I borrowed it from somewhere âover the
rainbowâ, a friendly reminder that small, readable code can be taken and
adapted.
There is a catch. You keep editing your code between runs. A diff
made last week may point at line 12, but line 12 is now something else.
patch does not check; it trusts the line numbers.
So a second function, _diff_applies,
checks first. For every line the diff expects to find, unchanged or
removed, it compares with the real source:
if src != content:
return FalseIf anything differs, the mutant is stale: it was made for code that no longer exists. The program marks it so, instead of running tests on nonsense. This pair is a good pattern in general: one function that does the work fast, and one that decides whether the work makes sense.
This is the most surprising part of the program. To test a mutant, the obvious way is to write the changed code into your file, run the tests, and put the file back. But then two mutants can never run at the same time, and a crash could leave your code broken.
mutation.py never writes to your files. Instead, it swaps the module in memory, inside pytest.
Look at the test command the tool builds:
PYTEST = "python3 -m pytest -p mutation --exitfirst --no-header --tb=no --quiet --assert=plain"The -p mutation part tells pytest to load mutation.py
itself as a plugin. So the same file is both the command you
type and an extension living inside pytest. Two functions are written
for pytest to call: pytest_addoption
teaches pytest a new flag, --mutation=<id>, and pytest_configure
runs before any test, and installs the mutant if that flag is
present.
install_module_loader
reads the diff from the notebook, applies it with patch,
and then runs the patched code into a brand-new module object:
patched_module = types.ModuleType(module_path)
exec(patched, patched_module.__dict__)
sys.modules[module_path] = patched_modulesys.modules is Pythonâs memory of modules already
imported. When a test later writes import yourmodule,
Python looks there first, finds the mutant, and never reads your file.
Each test run is its own process, so each one sees only its own
mutant.
The middle of the function works out the moduleâs import name from
its file path (is it mypkg.utils or
src.mypkg.utils?). That part is fiddly, and its comments
explain why: it tries every folder Python searches and keeps the
shortest name.
Now we can test a single mutant. Two functions do it.
run
starts pytest as a separate program and returns its exit code,
the number every program hands back when it ends (0 means success). Two
small details matter:
PYTHONDONTWRITEBYTECODE=1, so Python does not
leave cached .pyc files behind that could mix up runs.mutation_is_survivor
adds --mutation=<id> to the test command, calls
run, and reads the exit code like a verdict:
| Exit code | Meaning | What the program records |
|---|---|---|
| 0 | All tests passed with the mutant inside | Survivor |
| 5 | The plugin could not apply the diff | Stale |
| 4 | pytest did not know --mutation |
Stops with a hint: the plugin is not loaded |
| anything else | A test failed | Killed, nothing to record |
The exit code 4 case is a nice piece of care. Without it, a missing plugin would make every test run fail, and the tool would happily report âevery mutant killed, perfect tests!â. The code turns that silent lie into a loud error with a hint on how to fix it.
A real project can have thousands of mutants. One at a time, that could take days. So the program runs several at once, one per CPU core by default, minus one to keep your machine usable.
The helper is pool_for_each_par_map.
The name reads as: with a pool of workers, for each
item, parallel map. Think of a tailorâs workshop in
Dakar with four sewing machines. The owner does not cut all the fabric
for the whole month before sewing starts. She keeps exactly four pieces
in progress, and hands out a new one each time a machine is free:
if len(pending) >= limit:
done, pending = await asyncio.wait(pending, return_when=asyncio.FIRST_COMPLETED)
for task in done:
f(task.result())
future = loop.run_in_executor(pool, p, item)Each finished result goes straight to f, a callback that
updates the progress counter and writes to the notebook. Nothing piles
up in memory.
The same helper serves two very different jobs, with two kinds of pool:
Knowing when waiting and computing call for different tools is a lesson that goes far beyond this program.
play togetherWe now know every piece. mutation_play
reads almost like the five-step list from the introduction:
.mutations.ignored/.check_tests: are the tests green, and how long do they
take?play_create_mutations: make every mutant.mutation_exec: test every mutant.check_tests
runs your tests once, with coverage turned on, which is where the âwhich
lines are runâ data from earlier comes from. It also times the run. That
duration, called alpha in the code, becomes the patience
budget for every mutant:
timeout = alpha * 2If one mutant takes more than twice as long as the whole healthy
suite, something is stuck, and run gives up. If the tests
fail when run in parallel, the function does not give up: it tries again
one by one, and if that works, it warns you and carries on.
play_create_mutations
picks the files with your --include and
--exclude patterns (by default, every .py file
whose name does not contain âtestâ), and skips files no test touches.
Then it sorts:
items = sorted(items, key=lambda x: len(x[1]), reverse=True)Biggest files go first. It is the same reason you load the large stones into a truck before the gravel: if the biggest job starts last, every other worker sits idle while it finishes.
mutation_exec
first prints a worst-case time estimate, honestly labeled as such. It
loads the list of mutants a human already judged harmless, so they are
skipped. It can test only a sample, with --sampling=10% or
--sampling=100. Then it runs
mutation_is_survivor for each mutant through the thread
pool, and writes survivors and stale mutants into the notebook.
replayA list of 200 survivors is not useful on its own. Someone has to look
at each one and decide what it means. replay
is a small conversation loop for that.
It reloads the test command and seed saved in the notebook, so it
replays under the same conditions. It sorts survivors by the size of
their diff, largest first. Then, for each one, replay_mutation
runs the tests again. This matters: if you wrote a new test since the
last run, the mutant may now be killed, and it quietly leaves the list.
Only if it still survives do you see the diff and a menu:
Plus r to replay, s to skip to the back of
the queue, and q to quit. Your choices are saved after each
answer, so Ctrl+C loses nothing.
The âequivalentâ choice is special. It is not only stored in the
notebook, which play deletes each time. write_ignored_file
also writes the diff to a plain text file in
.mutations.ignored/, named after a fingerprint (a SHA-256
hash) of the diff, with your optional one-line reason on top. Because it
is a plain file, you can commit it to git, and your teammates inherit
your judgment. When the code changes so much that the diff no longer
fits, mutation gc and the start of every play
remove the outdated file automatically.
For scripts and other tools, the same actions exist as separate
commands: mutation survivors,
mutation inspect <id>, and
mutation classify <id> <category>, each with a
--json option. A program, or an AI assistant, can read
survivors and propose classifications, while a person keeps the last
word.
You have now read a complete, useful tool, about 2,500 lines in a single file, with no framework. A few small exercises, from easiest to hardest:
mutation play on a tiny project of your own with
three functions and three tests. Read every survivor with
mutation replay.min and max. Thanks to the metaclass, the
class alone is enough.InjectException
in full, and find a variable name its guesses would get wrong.All links in this post point to commit 74270f8, so line numbers stay correct even after the code moves on.
I have run mutation.py on 34 well-known Python libraries; the list is in DONE.md. Pick one you use, run it, and look at the survivors yourself. Most will be equivalent mutants or loose ends that do not matter; telling those apart from a real gap is the skill this post is about. If you do find a real one, take the time to understand it before opening an issue: maintainers deserve a careful report, not a drive-by.