Why AI checks fail silently: 1,278 restarts behind a healthy status
For three weeks a status command told me everything was fine. It said loggedIn: true. The credential behind it had expired on 11 July.
The service it authenticated had been restarting every few seconds the entire time. By the time I looked, the process manager had launched it 1,278 times. Nothing had told me, because the thing whose job it was to tell me was reporting that a credential existed, not that it worked.
That is a small bug with an expensive shape, and once I saw the shape I found it four more times in a week.
The one that should have been impossible
My always-on machine runs a health check every morning. It did its job. On two consecutive mornings it detected the outage and recorded three red statuses, including the one that mattered: the connector was down.
I never heard about it. The alert is sent with curl, and the reason the machine was broken was that it had run out of network sockets. The check could see the fire perfectly well. It just could not reach the phone.
Then it did something worse. It wrote its "already alerted" marker to disk. That marker is how it avoids nagging you about the same problem twice, and it was written unconditionally, after the send, whether or not the send worked. So the alert was not merely delayed. It was permanently retired, and because the fault signature never changed afterwards, recovery would not have pushed either.
A monitor that goes quiet when its subject dies is worse than no monitor. With no monitor you know you are blind. With that one, silence read as health.
The pattern, and it is not a machine problem
Neither of those is an AI story. They are just badly wired checks. But the same week, working with agents on my own notes and code, I hit the identical shape three more times.
A retracted claim reached a published post because the note it came from had been corrected in one place and not the other. A register that records what I have shipped told a review that no videos had gone out, four days after one had. Open work was counted by searching for keywords, which over-counted, because notes on finished work contain the word "blocked".
Every one of those parses cleanly. Every one is internally consistent. Every one was false. And in each case something was checking, which is precisely why nobody looked.
The unifying property is not absence of verification. It is verification that could not fail.
What the evidence says about the cost
I had assumed this was my own sloppiness. Then I went looking, and the research is blunter than I expected.
Researchers at Carnegie Mellon studied 806 repositories that adopted an AI coding tool against 1,380 matched ones that did not. In the first month, lines added rose by 281%. By the second month that was 48%. Averaged across all months after adoption there was no statistically significant change in commit volume at all.
The quality figures move the other way and stay there. Static analysis warnings up 30.26%. Code complexity up 41.64%. Duplicate lines up 7.03%. The authors describe the velocity gain as large but transient.
The speed is borrowed. The mess is kept.
One caveat I will state rather than hide, because it cuts against my own argument: their velocity measure is lines added, which is a weak proxy. More lines is not more work done, and a 281% rise in lines sitting next to a 41% rise in complexity may be one phenomenon counted twice. The decay is the robust finding. The size of the peak is softer than it looks.
Carnegie Mellon, 806 adopting repositories against 1,380 matched controls. The gain decays. The complexity does not.
The rule worth stealing
I found the sharp version of this in a conference talk by the chief executive of a code quality company, which means I read it sceptically. Five of his six quantified benefits came from his own firm and the talk ended at a booth. The framework survived that; none of the numbers should be repeated.
The part that survives is a design rule, and it is falsifiable, which is more than most conference content manages.
Zero-trust verification means using a different model. Every model has characteristic biases and blind spots, and they travel with it. So a critic running on the generator's model inherits exactly the weaknesses that produced the error, and returns agreement rather than verification.
Generalise it past AI and you get the sentence I keep coming back to. A check that depends on the thing it is checking is not a check. The auth command depended on the credential store it was reporting on. The health check depended on the network it was watching. The same-model critic depends on the reasoning it is supposed to audit.
The second useful distinction from the same talk: functional correctness is not quality. On his benchmark, current models pass "does it work" almost universally while still producing complexity, bugs and security holes. The thing agents are graded on is not the thing that hurts you in six months.
What I changed
Four things, and only one of them is clever.
Verification now runs in two layers. An algorithmic layer where the oracle is a command's exit code: tests, lint, type checks, a script that must print nothing. And an agentic layer for what no command can express: intent, business logic, whether the thing does what was actually asked.
The layers catch different defects. Neither substitutes for the other.
Every prompt I write for autonomous work now names the oracle for each layer, and forbids the verifying pass from simply continuing the generating pass. "Check it three times" describes how often. An oracle is what would prove it wrong. Only the second one is verification.
The health check writes its signature only on confirmed delivery, so a failed send retries instead of recording itself as sent.
And when a rule changes, the old wording goes into my consistency checker at the moment of the change, rather than after a stale copy causes damage. Every entry in that list used to be a scar. They should be inoculations.
Then I tested the checker, which is the part I would skip if I were you
I have a script that sweeps my notes for exactly this class of defect. It exits clean every Monday. Last week I realised I had no idea whether clean meant clean or meant blind, so I planted defects in a copy and ran it against them.
The first test was invalid. I put the planted files in a directory the script deliberately excludes, so it reported clean and I briefly believed I had found a serious bug.
The second test was invalid too. My fixture's filename contained a word the script treats as a negation, so the planted defect filtered itself out.
The third test worked, and found something real: the negation filter was matching file paths as well as file contents, so any file with "retired" or "corrected" or "never" in its name was invisible to that entire scan. No real file was hitting it, so it had never hidden anything. A latent blind spot, closed before it cost me.
Three attempts to verify one 80-line shell script. That is the honest cost, and it is why almost nobody does it.
And in case this reads as someone who has now got it sorted: an hour later I deleted a live DNS record before proving I could write the replacement back. The write failed. A domain sat with no email authentication policy for two minutes. I had verified the read path and assumed the write path, which is the same error in a different coat.
I do not think the answer is checking more. It is checking with something that does not share the failure. A different model for the critic. A command whose exit code you did not write in the same breath as the code it tests. A monitor with a path to you that survives the thing it monitors.
If you use AI to produce anything you later depend on, the question is not whether you have checks. It is whether any of yours has ever been shown to fail.
Mine hadn't. What would it take to find out about yours?