The credibility test we run backwards: ask who they were paying
Jamie Watters
Operational resilience and AI delivery practitioner.
Who this is for. Anyone weighing a warning from inside an industry, about AI or anything else, and trying to work out how much the messenger's position should count. By the end you'll have three questions to ask of any such warning, and the reason the usual test runs the wrong way round.
Skip it if you want the AI-risk argument itself. This is about how to weigh who is making it.
On the length. Twelve minutes, and it carries a correction: an earlier version of this test was wrong and the piece says where. The three questions are at the end and stand on their own.
In the same week, two AI researchers said roughly the same frightening thing in public. One of them quit to say it and gave up equity that had not yet vested. The other still had his job.
When this piece went up on Monday it said the podcast panel discussing them believed the second man and mocked the first. That was wrong, and this is the corrected version. I had the full transcript in my own files. I had not read all of it. The correction turned out to be more interesting than the thing I got wrong, so it is the spine of the piece now rather than a footnote at the bottom.
Jacob Coxon spent about three years on pretraining research, first at OpenAI and then at Anthropic. He told TIME he walked away on 8 September. The next day he posted that neither company is acting responsibly, that they are "racing straight to self-improving superintelligence and gambling with our lives".
Evan Hubinger leads Alignment Science at Anthropic. The same day, he replied on X: "Jacob is correct here, we really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade. I believe Anthropic is trying its best, but we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to."
Peter Diamandis's Moonshots panel covered both in one episode. Alex Wissner-Gross dismissed the resignation: a "well-worn tradition" of researchers "going out in a blaze of glory, including after being there for only a few months", and "I heavily discount any particular blaze of glory rage quit couched as virtue signaling. Don't give it much credibility."
Then, in the same turn, Wissner-Gross did this: "said that this 10% pdoom again I discount". And he gave a reason, a Fermi-paradox argument that a galaxy carrying a risk that size would already have been eaten.
So he discounted both. He was consistent, and Monday's version of this said he was not.
What each sentence cost, priced properly
Here is where the tenure point matters, and it cuts against me rather than for me.
Coxon had been at Anthropic four months. He told Axios that staff need six months for stock to vest, so he left two months short of the cliff: "I left before any of my equity vested." He keeps the equity from OpenAI, where nearly all of those three years were spent. So the receipt is four months of unvested options at a company he had just joined, not three years of anything, and it is not money he had earned. It is money he would have earned in eight more weeks.
That is a real cost and a smaller one than my first version implied, and Wissner-Gross's throwaway line about people who leave loudly "after being there for only a few months" was closer to the shape of the facts than I was.
Hubinger's statement has no findable cost. He was in the role the day before and the day after, and his employer's public position already ran in the same direction: his chief executive gave a range of ten to twenty-five per cent on a podcast in October 2023, for something going "really quite catastrophically wrong on the scale of human civilization", and told Axios in September 2025 there was "a 25% chance that things go really, really badly".
Two cautions on that. Amodei's range is wider in scope than extinction and carries no ten-year bound, so Hubinger's number is the sharper claim, not a restatement of his boss's. And "no findable cost" is not the same as "no cost": a serving alignment lead saying in public that his employer has no plan may well have paid something inside the building that none of us can see.
The test we think we're running
Ask people how they judge a claim and most will tell you they follow the incentives. Then watch which way they follow them.
A loud exit reads as drama. Staying reads as measured. The person who quit is assumed to be performing, and the person who stayed is assumed to be brave within the constraints. It feels like the sophisticated read. It is the incentive check run backwards, because the exit came with a receipt and the insider's sentence came free.
The bit that matters is narrower than "did they leave", and getting it precise is what makes it useful at work. The cost that counts is what saying it cost them at the moment they said it.
Your colleague who tells the sponsor in the meeting that the date will slip is paying, in that meeting, in front of the person it embarrasses. The same colleague writing it in a leaving email three weeks later pays nothing. Same words, same person, same slip. Only one of them is evidence.
Coxon didn't quit and then complain. He quit in order to say it, and the equity went with the sentence. That part holds even at four months.
Cost is only half a sum
Before any of that, the obvious objection, and it is the one that nearly sinks the test.
A cost only tells you something when you weigh it against what the person stood to gain.
Coxon forfeited four months of unvested equity. I do not know what that was worth. Neither does anyone else writing about this, mine included, and the figure is doing a lot of quiet work in all of our accounts. He kept what he had already earned at OpenAI. Against that he gained something real: attention, the respect of people who already agreed with him, and a permanent place in the story. Whether his sentence cost him on balance is a sum nobody outside his own accounts can do.
So the question is not what it cost. It is what it cost net, and the sharper version of that is: who was he paying? A cost is evidence when it is paid to the people who can punish him for it. A cost is theatre when the people watching hand him something back that is worth more.
That test is harder to run and it survives contact with the cases. Say the quiet thing in the meeting, in front of the person it embarrasses, and the audience is the one holding your review. Say it in a leaving email and the audience is the people who already agreed with you. Coxon lost the first audience and won the second.
And here is where I have to stop short of where I wanted to get to. Costly signalling only tells you anything when the cost falls harder on the people making the claim falsely than on the people making it truly. Nothing here establishes that. A researcher who is sincerely wrong can forfeit four months of options exactly as readily as one who is right. Naming the audience improves the question. It does not close it, and the people who work on this for a living say as much: measuring a cost at equilibrium is not a test of the signal.
So the receipt is real and it is thin, and thinner than Monday's version of this argument, which is the version most people are running.
What cost cannot buy
Here is where it stops, and this is the part both sides of the argument skip.
Cost proves sincerity. It does not prove accuracy.
Coxon giving up his equity means he believes what he said. It does not mean he is right. Sincerity bought at a price is exactly how a mistaken belief acquires a reputation for integrity, and the history of costly warnings that turned out to be wrong is long.
The reverse holds too. Hubinger's statement being free does not make it empty. He and his co-authors introduced the term mesa-optimisation in a 2019 paper, and he co-authored Sleeper Agents in 2024, which built proof-of-concept models that behaved deceptively in a way the safety training it tested did not remove. He spends his days building the tools that look for the failure he is warning about.
So cost sorts belief from posture. It does not sort true from false. Two different questions, and most of the coverage of that week collapsed them, treating "is he sincere" as though it answered "is he right".
As for the number itself, it has no method attached. No model, no reference class, and nothing named that would move it. It does have a horizon, a decade, which is more than most such numbers carry. It still forbids nothing you could go and look for next year. Hubinger was careful, to be fair to him, and said "I personally think". That hedge survived into most of the coverage. It is his belief, honestly labelled as his belief, and it is not a measurement.
Three questions for any warning
- Who were they paying? Not whether they are angry, not whether they left. Did the sentence cost them something with an audience that could punish them for it, or did it buy them something from an audience that was always going to applaud.
- Is the claim checkable, or only believable? Can you name the thing that would show it to be false.
- What would change their mind? Ask them. If the answer is nothing, you have a belief, not a forecast.
Run the two statements through it. Coxon's passes the first thinly: four months of options, forfeited in front of an employer, against applause from people who were already on his side. On the other two it is silent, because he has not said what would change his mind either. Hubinger's fails the second and third, and I suspect he would say so himself.
Sorting them tells you which is a belief with a receipt attached. It does not tell you which is right, and anyone who offers you a test that does is selling something.
For what to ask after the sort, I wrote a companion piece the next day, on reading the same exchange as a risk register entry: what has to sit beside a likelihood before anyone signs it, why the third sentence of Hubinger's post matters more than the number everyone argued about, and the four questions I would put to any percentage handed to me at work.
What this said on Monday was the easier and more flattering version: a panel that mocked the man who paid and swallowed the man who did not. What the transcript shows is one man who discounted both, on reasons he said out loud, one of which was better than mine. I would rather have the transcript.
Sources
- Evan Hubinger (@EvanHub), X, 9 September 2026. The quote is reproduced identically by Dexerto, Wall St Engine and BigGo Finance, each linking the original post.
- Jacob Coxon (@hilbertspaess), X, 9 September 2026.
- TIME, "He Helped Build Powerful AI at OpenAI and Anthropic. Now He's Afraid It Could Kill Us", 9 September 2026, for the 8 September departure and the three years across both labs. Corrected 15 September: an earlier version of this piece cited TIME for the forfeited equity and for his leaving the field. TIME carries neither.
- Axios, "Scoop: Anthropic whistleblower gave up his equity to leave the company", 9 September 2026, for the four months at Anthropic, the six-month vesting cliff, the retained OpenAI equity, and his own words: "I left before any of my equity vested."
- Axios, "Amodei on AI: 'There's a 25% chance that things go really, really badly'", 17 September 2025. The wider ten to twenty-five per cent range predates it: Amodei gave it on the Logan Bartlett Show in October 2023, for something going "really quite catastrophically wrong on the scale of human civilization", which is broader than extinction and carries no ten-year bound.
- Hubinger and colleagues, "Risks from Learned Optimization in Advanced Machine Learning Systems", 2019, which calls mesa-optimisation "a neologism we introduce in this paper", and "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", 2024, which tests proof-of-concept deceptive models rather than showing the behaviour arises by default.
- Moonshots with Peter Diamandis, "Three Lab Warnings in Five Days", September 2026, read as a full transcript. Alex Wissner-Gross both dismissed the resignation and, in the same turn, said of the estimate: "said that this 10% pdoom again I discount."
- Lachmann, Számadó and Bergstrom, "Cost and conflict in animal signals and human language", PNAS, for the point that a cost measured at equilibrium is not itself a test of a signal.
Substantially revised 15 September after five independent reviews. Four of them found the panel claim, which was the piece's central example and was wrong. What changed and why is in the review assessment filed with the draft.