The check had been green for a week. It greps every diff under src/ for import mock, because production code has no business importing a mock library. Then it went red on src/payments.py, a file with no mock import anywhere in it. The word mock was sitting in a docstring, in a sentence telling the next developer not to import one. The grep matched the sentence. The quickest way to clear that red is to delete the check.
The series opener named the loop that runs generate, check, steer, retry, stop, and put the hard part on the check that decides good enough, stop. Last week's piece built that check as no-mocks.sh and set a gate beside it, a hook that refuses a bad write instead of catching it after the fact. This one stays with the check and asks what happens when it is wrong: how to tell a broken instrument from a signal that was never there.
What a check is here
A check is a function from the agent's output to a verdict: pass or fail. In the loop it is the thing that decides whether an iteration is acceptable and whether to run another. A check comes in two kinds. A deterministic check runs the code or scans the text and returns the same verdict every time: grep, pytest, an exit status. A model-graded check asks another model is this good? and reaches criteria the first cannot express, at the cost of the same unreliability the loop was built to contain.
This piece is about the deterministic kind, because it is the kind you can hold in your hand and debug. When a deterministic check gives you a verdict you disagree with, the verdict is wrong in a way you can inspect line by line. That is exactly the property the argument below leans on.
Two readings of the same quiet
A check that comes back quiet is telling you one of two things, and it does not tell you which:
- Absent signal. The thing the check guards against genuinely is not there. This is the good news you wanted.
- Broken instrument. The thing is there, or would be there, but the check cannot see it. The guard is off and the diff looks the same as if it were on.
A false alarm, the check firing on a clean file, is the visible failure. It is annoying, but it announces itself; you know the instrument is off because it is screaming at nothing. The dangerous failure is the opposite one, and it is silent: a check weakened until it stops seeing real violations returns the same green as a check that correctly found nothing. You cannot feel the difference between "safe" and "blind" from the verdict alone.
So the misfire is really about the check, not the file it landed on, and it sets a trap worth naming early. A red mark on a file you know is clean reads as a reason to stop trusting the check, and it tends to show up on the very run where that same check is catching something real elsewhere. Look at what the pattern actually matched before you act on the verdict it returned.
The misfire, up close
Make it concrete with the check itself. The version that misfires is the naive one, a bare substring match, before the ^ anchor last week's piece put on the pattern:
grep -rn 'import mock' src/ && exit 1
For a week it did its job. Then you added src/payments.py, whose docstring happens to document the rule the check enforces:
"""Charge processing. Do not import mock here, use the fakes in tests/."""
import stripe
There is no mock import in that file. Run the check and it fires anyway, alongside a real leftover it correctly caught in src/badcache.py:
$ bash no-mocks.sh
src/payments.py:1:"""Charge processing. Do not import mock here, use the fakes in tests/."""
src/badcache.py:1:import mock # leftover from a spike
The badcache.py line is a real catch, an import mock that has been sitting in production code. The payments.py line is the misfire: that file is clean, and the check flagged it because import mock appears as a substring inside prose documenting the very rule the check enforces.
The tempting fix, and what it costs
You have a red check on a clean file. The path of least resistance has three forms, and all three are the same move:
- Delete
no-mocks.sh. The red goes away. - Weaken it: add
--exclude=payments.py, or only run it on Tuesdays, or drop it to a warning nobody reads. - Conclude the check was never worth much: "grep is dumb, mock imports are rare anyway, this is more trouble than it catches."
Every one of these makes the noise stop, and stopping the noise feels like fixing the problem. Re-read the output. The same run that misfired on payments.py also, correctly, caught the real import mock in badcache.py. Delete or defang the check and that catch disappears with the false alarm. The next real mock import to land in src/ sails through, and the suite stays green, because green is now what this check returns for everything.
That is the inversion. The misfire tempts you to conclude the signal is not worth watching, at the exact moment the same check proved the signal is real. You would be reasoning about the world from the failure of your instrument to see it.
Fix the instrument instead
The misfire told you something specific: the pattern matched prose, not code. So make the pattern match code. That is the one-character change last week's piece already carried, the ^ that anchors the pattern to the start of a line, where an import statement lives and a docstring sentence does not. Add it and payments.py goes quiet while badcache.py still fails.
Both make the red on payments.py disappear, and only one keeps catching badcache.py. Deleting the check clears the false alarm by throwing away the real catch with it; anchoring the pattern clears the false alarm and leaves the real catch standing.
I run the same shape of check on my own agent's replies. A deterministic scan looks for a short list of banned phrases: a placation opener, or an unbacked the work is verified when nothing was actually run. One day it fired on a reply that was clean. The banned phrase was sitting inside a quotation of the rule that bans it, and inside a backticked identifier the agent was naming rather than using. The check had matched the words, not their use, the same way the grep matched the docstring. Loosening the phrase list until that alarm stopped would have let the next real unbacked claim through, so I scoped the matcher instead. It blanks the cited spans, the backtick spans and quoted strings and negated clauses, before the scan runs, so an empty the work is verified still fires while a citation of the rule goes quiet.
The anchored check still is not a perfect mock detector. An aliased import unittest.mock as m is a different pattern, and an indented import inside a function slips past a start-of-line anchor. A real linter closes both without any of this hand-rolling: flake8-tidy-imports carries a banned-modules list, and ruff's F401 flags imports the code never actually uses. If mock-in-production is the exact thing you are guarding, reach for one of those. The grep still earns its place as the first thing you write, because it is the cheapest deterministic check a team reaches for in a hook, one line in a script that already runs. And the lesson holds after you upgrade the instrument, because a linter rule is a pattern too. F401 matches what looks like an unused import, not the fact of use, and it has edge cases where it reads the text wrong. Every one of those is a calibration reading you use to sharpen the rule, never a reason to stop measuring.
The rule that generalizes
A misfire is data about the instrument, never a verdict about the world. When a deterministic check fires on something you know is clean, the finding is "my check matches too much," not "the thing I was checking for does not exist."
The asymmetry is what makes the discipline worth holding. A false alarm is loud and self-correcting: it interrupts you until you deal with it, and it costs you the few minutes the fix takes. A deleted or defanged check fails the other way. It goes quiet and stays quiet: no output, no red, no interruption, just a signal nobody is watching anymore, indistinguishable from a signal that was never there. Every real violation that lands afterward passes clean, and nothing on screen tells you the guard is gone.
This was one of the questions the series opener left open: how to tell a broken instrument from an absent one. Its sibling piece took another, the gate that refuses a bad write instead of catching it a beat too late, and the line between a rule that observes and one that refuses. The question underneath both is still open. Every check and every gate is fed by rules the agent carries as loaded text, paid for on every turn whether they apply that turn or not. What an instruction costs to keep loaded, and what changes when you load it only where it applies, is where the series goes next.
I work on Reporails, deterministic diagnostics for the instruction files, rules, and prompts that steer coding agents. It reads the steering surface and tells you, with measured evidence, where it drifts. The checks here are the same idea pointed at a diff: what a rule actually matches, held up against what you meant it to match.