Deleting a safety rule on purpose to see whether a test notices

What question it answers

A test count tells you how much was written. It does not tell you what would have to go wrong for a test to stay green, and that is the question a sceptical reader is actually asking.

So the kit answers it by breaking things on purpose: each mutation removes one rule this project makes in public, runs the suite that is supposed to defend it, and names the test that went red.

What it does

`npm run falsify` ran here on 2026-08-09 and printed, verbatim: "5 caught, 0 survived, 0 stale, in 2.2s" and "Every rule tested here fails loudly when removed." Each row named the rule and its witness — `approval-actor-guard` caught by "approval workflow rejects an agent pretending to make the human decision"; `approval-threshold-boundary` by "renewal at threshold requires approval and records the human decision"; `webhook-signature-verification` by "webhook verification is constant-time, replay-bounded and fail-closed"; `definition-version-immutability` by "persisted definition versions: registration is idempotent, drift fails loudly"; `managed-fields-public-write` by "a fully managed module generates no public write, and a mixed one does". A sixth, `delivery-cost-rounding`, printed as "not run (slow — run with --all)" rather than being quietly dropped from the set.

Three outcomes, two of which are failures. `caught` means the suite went red and the report names the test. `survived` means the suite stayed green with the rule removed — a gap in the tests. `stale` means the code the mutation aims at is gone, so it proved nothing, and that is a failure on purpose: a falsification kit that quietly stops aiming at anything is worse than not having one, because it goes on printing reassurance. The run exits non-zero on either.

Baselines are measured rather than assumed: every target suite runs unmutated first. A suite that is already red would make every mutation look caught, which is the one way this tool could lie in the flattering direction.

Each mutation carries the claim ids it defends. `--json` emits `falsifyContract: 1` with, per mutation, the file, the exact source string removed, the defending test, the rule, what breaking it would mean, the claim ids, how many occurrences were found against how many were expected, the outcome and the name of the catching test — plus a `skipped[]` array carrying the slow mutation and its reason. The `elapsedMs` field on the JSON runs here read between 2,110 and 2,385 ms, which is the honest precision available for a wall-clock number.

What it already found is the reason to keep it. `managed-fields-public-write` survived the first time it ran: removing the read-only rule from the module factory left the factory's own test green, and only the end-to-end suites noticed. A rule about generated output should fail in the test that reads the generated output, so a direct assertion was added. The interesting output of a falsification run is not the row of ticks.

What it refuses to claim

This is the second of the two tools in this cluster that publishes no `limitations[]` array. Its machine-readable self-reporting is different in kind: the `stale` outcome, which is how the kit reports that it has stopped aiming at anything, and the `skipped[]` list, which is how it reports that the default run is narrower than the full set. Both are in the JSON, and both fail or annotate the run rather than being silently absorbed.

The prose boundaries are three, and they are stated as flatly as anything in this repository. It is not a mutation-testing tool: it does not generate mutations, sample them or compute a mutation score, and six hand-written mutations aimed at six claims is a different instrument from a coverage metric. It does not prove the rules are correct, only that they are held by a test — a wrong rule, faithfully defended, passes every mutation here. And it covers six rules, not the ledger: the claims ledger carries twenty-two claims, each mutation names which ones it defends, and the rest rest on their own evidence and are not falsified by this command.

It does not tell you which tests are missing. It tells you that these five specific rules currently have a witness, and it will tell you loudly when one of them stops having one.

What it costs to run

About 2.6 seconds for the default set here, exit code 0. The full set with `--all` includes one mutation whose only witness takes roughly a minute.

It edits real source files in place, one at a time, and restores them in a `finally` with the restoration verified byte-for-byte. If a restore ever fails it throws immediately, naming the file and the command that fixes it, rather than letting the run scroll it away.

The safety property is a precondition, not a copy: it refuses to start if any target file has uncommitted changes. So even a killed process leaves damage that a single `git checkout -- <file>` undoes completely. A scratch copy of the tree would avoid the edit entirely and cost a full copy per run; the git precondition buys the same assurance for the price of one `git status`.

Where it fits

It sits beside the quality gates rather than inside them: the gates say the tests must pass, and this says what the tests would notice. Section 4 of the gates lists mutation tests for versioned policies as a production gate that does not exist — this kit is not that, and does not claim to be.

The five rules it currently defends are the ones where being wrong costs the most: an agent recording a human's approval decision, a deal exactly at the approval threshold, a forged signature webhook, a policy version edited in place so every historical decision citing it now cites something else, and generic CRUD writing fields only a trusted action may set.