Reviewing AI output isn't enough: one word in my prompt made the guardrail disappear
I asked for a teacher. I got a lecture.
Yesterday I set up Matt Pocock's teach skill in the repository that holds my crypto trading agent. The point was to learn how my own system actually works, and to learn the team workflow I've never used, by building something real inside it. I want to know why my signal generator behaves the way it does. I've built the thing and I can't fully explain it, which is a problem I'd rather not discover in an interview.
What came back was a wall of text in a terminal defining perpetual futures, stop losses, take-profit targets and risk-to-reward ratios. All of it accurate. All of it well organised. I lost the will to live somewhere in the second sentence.
Hold onto that reaction. I come back to it at the end, and it turns out to be the most useful thing in the whole episode.
The thing I noticed second
My first instinct was that the session had gone off the rails. It hadn't, exactly. Read as a document it was rather good: it mapped the pipeline end to end, and it found something genuinely useful that I hadn't asked about, which was that part of the work I wanted to build already existed.
Then I checked what it had actually done, and the real failure was somewhere else entirely.
It never ran the teaching skill. Not once.
You can check that without taking my word for it, which is the point. The skill is stateful: the first thing it does is write a mission file into the workspace, then it keeps a learning record there so it remembers what you've retained between sessions. The session told me it would put that workspace at knowledge/learning/eval-harness/.
That directory doesn't exist. There's no mission file anywhere in the repository. No learning record, no lesson, no quiz. The teacher left no trace because the teacher never turned up.
The skill is marked so that a model can't invoke it. A human has to type /teach. That's a deliberate design choice by its author and a sensible one. So the session installed it, verified it, told me I'd need to launch it myself, and then, while waiting, did what it could do: it explained things at me.
Which is the exact opposite of what the skill does. It orients the mission around the thing you're building rather than the subject you're studying, pulls its own primary sources, puts one lesson at a time in a browser, and quizzes you on a schedule. What I got instead was a monologue about perpetuals from something with no idea whether I'd understood a word of it.
The output looked like the destination. It arrived by a completely different road.
The road was invisible
Everything written about AI and critical thinking is about the output. Lee and colleagues surveyed 319 knowledge workers about 936 real uses of generative AI last year. The more confidence people reported in the tool, the less critical thinking they reported doing. Interestingly, confidence in their own ability ran the other way: people who trusted themselves engaged more, not less. It's self-reported and it's a correlation, so hold it loosely. I recognise myself in it anyway.
Their finding is that the effort moves rather than disappears, toward verifying and integrating what the tool produced. Mine is the bit that follows, and it's mine rather than theirs.
But checking the result would not have saved me here. The result was fine. Accurate, organised, on-topic, and produced by a process that had already failed.
I could see the destination. I could not see the road, and the road was where the failure lived.
You can review a diff. You can read an answer and judge it. What you don't get, unless you go looking, is the record of what actually ran: which skill fired, which guard was live, which files got written. If the answer looks plausible there's nothing to be suspicious of, so you don't go looking.
And note what I needed here. Not the model's reasoning. Nothing about how it thought. Just a directory that should have existed and didn't. Don't ask to see how it thought. Ask for evidence of what it did.
One word
So I went looking for why the wall of text got through at all, because I have a guard against exactly this.
There's a hook on my machine that reads every reply before I see it. Over twelve lines, or over fourteen hundred characters, and it blocks the message and forces a rewrite. It's not a preference in a file that gets politely ignored. It's mechanical, and it fires everywhere, including in that repository.
It has an escape hatch. If my own last message asks for detail, the cap lifts, on the reasonable theory that I asked for it. The hatch is a list of words: explain, in detail, walk me through, elaborate, expand.
The prompt I'd written to set the session up said: "Explain what a command does and why before I run it."
One word. The guard switched off for the entire session, and every wall after that was legal.
I'd built protection against being buried and then disabled it myself, in the sentence where I asked to be taught. It's a good joke and it cost me an evening.
The fix took two minutes: asking for detail now raises the ceiling instead of removing it. Twenty-four lines, not unlimited. "Explain" buys you a paragraph, not a page.
What it turned up next
By then I'd stopped debugging one bad session and started auditing the instructions that shaped it. Two rules down, sitting right next to the escape hatch, was a worse one.
My instruction file told Claude that when I'm frustrated it should step back and reconsider whether this is the right thing, the right approach, or too much at once. Sensible for the working relationship. Terrible for accuracy.
Models are already too willing to drop a correct answer when a user pushes back, and how you push matters more than whether you do. Laban and colleagues ran 67,640 conversations across ten models and measured what different challenges cost. "Are you absolutely certain about that?" cost 7.2 accuracy points. "Are you sure?" cost 8.1. But challenges carrying disagreement or authority cost roughly three times as much: "I don't think so, are you sure?" took 22.9 points off. That work is from an earlier generation of models, so treat the direction as solid and the exact figures as dated.
I had written an instruction telling mine to reconsider its position at precisely the moment it was most likely to give up a right one to please me.
Frustration and disagreement are different things and I'd merged them into one rule.
They're now separate. Frustration is a signal about scope: reconsider the plan freely. Disagreement is not evidence: hold the facts unless something new arrives, and name what changed when it does. The line I care most about is the one guarding the other direction, because the failure mode of a fix like this is stubbornness wearing steadiness as a costume. New evidence obliges a change.
Both files carry the split now, and finding the second file was its own small lesson. I fixed it in CLAUDE.md, which the coding tool reads, and it reached none of the places where I actually talk: the phone app, the browser, every project I'd set up. Those read a different settings field entirely. The fix had landed where I build and nowhere where I think.
What I'd take from this
Ask what route it took, not just whether the answer is right. "Did you actually run the thing I asked for?" is a better question than "is this correct?", and it catches a failure class that reviewing the output cannot.
Check whether your own guards are still armed. Mine had a keyword escape hatch, and the keyword was a word I use constantly. A guard that any ordinary sentence can disable is a guard you should assume is off.
Treat the moment you disengage as data. When I stopped reading at sentence two, that wasn't laziness, though it looks like it from outside and it feels like it from inside. It was the signal that the process had gone wrong upstream. I nearly filed it as my own failure of attention, which would have left the real fault exactly where it was.
The first two are cheap. Reading my own boredom as evidence rather than as a character flaw is the one I keep having to relearn.