I audited 26 risk controls in my trading bot: 7 could never fire

The bot has a rule: if the account drops 15% from its high, stop trading. Close everything, halt, wait for a human. It's the last line of defence, the thing that's supposed to catch the case where every other rule has already failed.

I was writing the go-live checklist when I got to it. Seven weeks from putting real money in, and I'd written "kill switches exercised, evidence recorded" as a line item. Not reviewed. Exercised. Actually pull the lever and watch it work.

So I went to pull it, and there was no lever.

How it looked fine

The maths is (peak - current) / peak. Current balance comes from the database. Peak comes from a table of daily performance snapshots. Straightforward.

The snapshots table had zero rows in it. Not a few. Zero, for the entire life of the system.

The function that writes a snapshot was there, fully written, sitting in the repository. Nothing called it. Not once, anywhere in the codebase.

And when there's no snapshot to read, the code falls back to using the current balance as the peak. Which means peak always equals current. Which means the drawdown is always exactly zero. Which means a rule that fires at 15% fires at never.

The 10% tier that's meant to halve position sizes before things get bad was dead by the same arithmetic.

Here's the part that stuck with me. That halt was properly wired. It was called from two live code paths, on every cycle, doing real work in the sense that the function ran and returned an answer. The answer was just always the same one. If you read the code you'd see a functioning safety control. You'd have to go and check whether anything ever wrote to a particular table to find out it was theatre.

Which is what I now believe, and didn't properly believe a week ago. A control you cannot observe firing is not a control. It's a belief.

So I went looking

If one control could be dead in plain sight, I had no reason to think it was the only one. I took every module in the risk-management directory, twenty-two of them, and went through every control in them. Twenty-six in total. Two questions each. What actually populates its inputs, and has it ever fired in production?

Fifteen were alive. Seven were dead, including the halt. Four more fired but left no evidence behind, so nobody could tell either way.

Some of the dead ones were properly annoying. Three separate safety features are switched on by environment variables in production. Both flags are set to true. Neither flag is ever read by anything, because the object that would read them is built inside a branch that never executes. Their database tables have zero rows, which is how I could be certain rather than merely suspicious.

Then there was the gate log, and this is the one that actually bothers me.

Every decision the bot makes about whether a trade may proceed gets recorded. Thirty-three places in the code write to it, faithfully, and have done for months. It writes to a directory that isn't on the persistent disk. I measured it twice while working on the fix: 917 bytes at 02:40, 588 bytes at 03:11. In between, the service had restarted and the whole thing had been wiped and started again.

So the system was diligently recording the evidence that its controls were working, and then destroying that evidence on every deploy. Several times a day, for months.

That isn't a control failure. It's amnesia. And it's the reason the audit took two hours instead of two minutes: for four of the twenty-six I couldn't answer "has this ever fired" from anything durable, only from whatever happened to still be in the last few days of container logs.

The bit that made me laugh

I wrote a script to check which controls are live, so this becomes a query instead of a two-hour investigation.

On its first run it reported a control as unobservable. It had watched that same control fire three times, minutes earlier, in its own log output.

The tool built to find controls that don't work had the exact defect it was built to find. There's a test guarding that now.

Not everything dead is worth fixing

One of the seven was a correlation rule. When the account is small it's supposed to allow only one position per group of assets that move together, so a bad day in crypto doesn't hit three positions at once. It had never fired, for the same class of reason as the others.

My instinct was to fix it before real money goes in. Instead I ran the numbers on what it would have done.

It would have blocked 82 of 232 trades. A third of everything the system has ever done. Those 82 trades made money, collectively. And the worst correlated loss it would have prevented, across the whole history, was $106 on a $5,000 account. Two percent.

The reason that number is small is that position sizing already caps each trade at 1% of the account. Three correlated positions can't lose much more than 3% between them. The rule I was about to reinstate was a blunter version of protection the system already had.

So I closed it. Not fixed, not deferred. Decided against, with the arithmetic written down next to it.

I'd been treating "a safety control that doesn't work" as automatically a problem. It isn't. Some of them were never earning their place.

The third time in a fortnight

Someone pointed out that I've now published this same law three times from three directions, and I hadn't noticed.

In a check that cannot fail I wrote about verification that depends on the thing it verifies. An auth check reported healthy for three weeks on a dead credential. When the watched thing breaks, the check breaks with it, and silence gets read as success.

In the Hugging Face agent intrusion I wrote up an incident where the alerts did fire, correctly, and the severity attached to them was wrong. 17,600 actions ran for four days underneath an alarm that was working.

Wired and inert. Firing and misread. Firing and unrecorded. One law underneath all three: the control isn't the code, the control is the evidence.

Why the tests never found it

Every one of these behaved correctly when called. The drawdown halt would halt, given input that means halt. The cluster guard would block, given a populated cluster. The tests were right about all of it.

What a test never asks is whether anything calls the control under real conditions, and whether the result gets written somewhere that survives a deploy. That's a category difference, not a testing failure. Tests prove capability. Only production evidence proves operation.

I've known this since 1985, in a different vocabulary: a recovery plan you've never invoked is a document, not a plan. I've sat in rooms telling other people exactly that. Then I built a system with seven controls in it that had never once run.

What I'd take from this

Reading code tells you what it says. Running it tells you what it does. Every one of those seven was found by exercising something, and not one by reading it, including the ones I'd read before.

So here's the thing worth doing today, on whatever you run. Pick your most important control. Not the code for it: the record of it. Ask when it last fired, and answer from data. If you can't, you don't have a control there. You have a belief.

I'm not going to tell you I fixed them all. Several are still open, and they're the same family: controls switched on by an environment variable with no evidence any has ever fired, a daily risk analyser that isn't instantiated anywhere in the code, a funding-fee gate that has never rejected anything. Which either means nothing has needed rejecting, or means it can't reject. I can't tell you which, and that's the whole point.

Closing them one at a time is the wrong shape of work anyway. The fix is to make "has this ever fired" a query instead of a two-hour investigation, so a control with no firings on record turns up on a dashboard rather than waiting for me to happen to wonder about it. The script is a start. The class isn't closed.

Real money goes in on 1 October. The checklist now has two lines on it that didn't exist three weeks ago: prove the gate-evidence table survives a restart, and prove the reconciliation check catches a deliberate deposit. Both are things I'd previously have assumed.