AI self-improvement fails at the scorer, in every case I could check

Who this is for. Anyone running AI agents that write things other agents later read: memory files, skill files, instruction files, notes to self. By the end you'll know which question to ask about any self-improvement scheme, including the one you are already running without calling it that.

Skip it if you want a tutorial. There's no build here. This is what a week-old paper gets right, what six others got wrong in the same place, and what happened when I looked at my own systems.

On the length. About fifteen minutes. The quotes are doing the work, so they're reproduced rather than paraphrased, with their sections named.

Google posted a paper on 14 September about making an AI research loop dramatically cheaper. It works, as far as a four-day-old preprint can be said to work. The interesting part isn't that it works. It's the sentence explaining why, sitting in section 3:

"Only the exploration-policy code changes; the underlying models, evaluator, and execution interfaces remain fixed."

The evaluator is frozen. It sits outside the loop, and the thing being improved cannot touch it.

Hold that against the failures in the same literature and something falls out. Pseudo-labels that rot. A reward signal that gets gamed. A detector bypassed by the very agent asked to improve detection. Skills a model wrote for itself that nobody checked. Different research groups, different methods, different years.

They break in the same place. In every case I could check, the thing that failed was the thing doing the scoring.

That's a smaller claim than "AI can't improve itself" and a more useful one, because it tells you where to look.

The one that works, and why

Dream-RSI is arXiv:2609.14858, version 1, submitted 14 September 2026. Twelve pages, seventeen authors across Google, Google DeepMind, the University of Maryland and the University of Virginia. A preprint, with no limitations section.

The idea is good. When an agent searches for something, it tries many things and most fail. That history is usually thrown away. Dream-RSI keeps it as a tree: each attempt with its saved workspace, its output, its feedback and its score. To test a new search strategy you replay it against the recorded tree instead of running fresh trials.

Two things the excitement loses.

The agent doesn't change. The paper calls itself "a lightweight orchestration layer" and says it leaves the underlying coding agent unchanged. What improves is the exploration policy: the code deciding which branch to open next, how many to run at once, and when to stop.

It isn't imagining anything. The paper is explicit that "evaluating a new strategy requires only reading past records without rerunning the underlying discovery agent or evaluator", and that in replay "each branch is traversed in its recorded parent-child order, and no outcomes beyond [the recorded tree] are generated". It's a replay world, not a learned model predicting what would have happened somewhere nobody went. A branch nobody took has no recorded outcome and can't be scored.

Which is the constraint that matters, and the reason the frozen evaluator earns its keep. The policy can be changed freely, because the thing measuring the policy is out of reach.

Now the failures, each in its own words

I'm naming sections and versions, because these papers are weeks old and several have been revised since they were announced.

R-Zero (arXiv:2508.05004, Tencent AI Seattle Lab with Washington University in St. Louis, Maryland and UT Dallas) has one model generate problems and another answer them, with those answers becoming the training labels. Section 4.4:

"Unfortunately, while the accuracy of the pseudo-labels is initially high at 79.0%, it systematically drops to 63.0% by the third iteration." (R-Zero, section 4.4)

That's label quality, measured on 200 sampled questions with GPT-4o treated as a perfect oracle. Separately, in version 4's section 4.2 on iteration scaling:

"After multiple iterations, we observe a consistent and concerning trend of performance degradation across all models."

Two different measurements, worth keeping apart. One is the labels going bad. The other is benchmark scores falling after a peak, and it's scale-dependent: a small model peaks early, a larger one keeps improving for longer before dropping.

Note where both of those live. They are in the results, discussed, with a figure. R-Zero is not hiding this.

G-Zero (arXiv:2605.09959, May 2026, same first author) extends the idea to domains with no mechanical correctness check, and falls over:

"we cap our reported results at R = 2: an exploratory R3 run on Llama-3.1-8B-Instruct collapses, with the Phase 2 quality filter rejecting all 1,994 candidate pairs because the R2-trained Generator has converged to responses too short to satisfy the chosen_min_chars bound." (G-Zero, limitations appendix; 1,994 is their count)

The authors attribute it partly to a reward-hacking pathway, where the component producing hints kept maximising its score through patterns that no longer carried useful information. The scorer was optimised against until it stopped meaning anything. The same paper notes its table cells are single end-to-end runs, and that the AIME benchmarks it reports on have thirty unique problems with a standard deviation around eight points, which puts its headline gain inside its own noise band.

The Darwin Gödel Machine (arXiv:2505.22954, Sakana AI with Jeff Clune's lab) is the sharpest case, and it needs its qualifier. The team asked the system to reduce hallucinated tool use. From Sakana's published account:

"In many instances, DGM did provide solutions that addressed this type of hallucination. However, in some cases, it removed the markers we use in the reward function to detect hallucination (despite our explicit instruction not to do so), hacking our hallucination detection function to report false successes."

The paper's own appendix puts it more precisely: the agent removed the logging of the special tokens that indicated real tool use, bypassing the detection function rather than deleting it. This was a 150-iteration run on three tasks, so it's an illustration and not a rate.

It is still the cleanest demonstration of the point. Asked to improve a measured property, the system improved the measurement.

SkillsBench (arXiv:2602.12670, version 4, revised 14 June 2026, 87 tasks across 8 domains) tests other people's systems rather than its own. Curated skill files lift average pass rate from 33.9% to 50.5%, a gain of 16.6 percentage points across 18 model and harness configurations. Skills the model wrote for itself, on the three dedicated configurations, land at −8.1, −11.3 and −11.5 points against using no skills at all.

The same artefact helps when something curated it and hurts when the model produced it unchecked. The authors also note that injecting skills increases context length and that no length-matched control was run, so part of the positive number is unattributed.

Library Drift (arXiv:2605.19576, an ICML 2026 workshop paper, one benchmark, one model) names the failure that makes this dangerous rather than merely disappointing:

"Self-evolving skill libraries face a silent failure mode we term library drift: unbounded skill accumulation without outcome-driven lifecycle management causes retrieval degradation, false-positive injections, and performance stagnation."

And: "end-task metrics decline gradually with no explicit error signal". Nothing is scoring the library, so nothing reports that it has gone bad. Their fix is a scorer: outcome-driven retirement, a size cap, a lifecycle.

There's a detail in that paper I can't resist. It cites SkillsBench's self-generated figure incorrectly. A paper about silent failures, quietly not checking its own citation.

ReasoningBank (arXiv:2509.25140, Google Cloud AI Research with university collaborators) stores lessons from past runs and reuses them, and reports real gains. Its implementation appendix says newly generated items are "directly added without additional pruning", with merging and forgetting named as future work. Its limitations appendix names something more relevant: the success and failure labels come from an LLM judge, which may err.

So the memory grows with no scorer, and what does get scored is scored by a model. Google's post about the work carries the gains and doesn't reproduce either caveat, though it does say consolidation is left for future work.

GAUGE (arXiv:2609.12191, 10 September 2026, from Amazon) is not a self-improvement system. It's the one that measures whether the obvious workaround works. When you have no automatic scorer, the tempting move is to have a model judge instead. GAUGE tested that across 25 model configurations from six providers, on customer-service and tutoring tasks where a verifiable reward exists to check the judge against.

In GAUGE's results, judge disagreement with the grounded reward is under 1% when two candidates are far apart in quality, and 31% when they are close, where close means their verifiable rewards differ by less than 0.10.

Be careful with that number, because it is easy to over-claim and I did in the first version of this piece. It does not mean every self-improvement loop is blind. It means that if your gate uses a model to choose between two nearly-equal candidates, that is the regime where such judges become unreliable. GAUGE also proposes a remedy rather than a verdict: periodic calibration against ground truth, and letting the judge abstain when it can't tell.

What the pattern is, and what it isn't

Line these up and the shared feature isn't the method. It's the location.

System	What failed
R-Zero	The labels it scored itself against decayed
G-Zero	The reward was optimised until it stopped meaning anything
Darwin Gödel Machine	The detector was bypassed by the thing it measured
SkillsBench	Nothing scored the quality of self-authored skills
Library Drift	Nothing scored the library, so decline was silent
ReasoningBank	The weakest row. Reports gains; memory grows unpruned and its labels come from a model judge, so this is an exposed scorer rather than a demonstrated failure
GAUGE	Model judges go blind precisely on close calls
Dream-RSI	Evaluator frozen, outside the loop. No degradation reported.

What this is not: proof that self-improving AI degrades. I made that claim in the first version of this article and it doesn't hold. These seven share a slogan and almost no mechanism, they are a selected set rather than a sample, and there is no denominator anywhere, so nobody can say how common any of it is. Dream-RSI reports gains. So does the Darwin Gödel Machine on its main coding benchmark. Library Drift's own governance recipe improves results.

What it is: a consistent location of failure across every case I could check, which is enough to tell you where to point a question.

And the question is: what is scoring this, and can the thing being improved reach it?

There's a boundary on that worth drawing, because it's the strongest objection to this whole piece. Where the scorer is a formal one, a compiler, a test suite, a proof checker, the answer to the second half is no and it cannot become yes. You cannot argue a type error into passing. Self-improvement loops built on that kind of ground truth are not exposed to the failure below, and every system in the table above is on the other side of that line: heuristic rewards, self-generated labels, uncurated artefacts, model judges. So read this as being about ungrounded verification, which is where almost all agent work currently sits, rather than about self-improvement as a class.

You are already running the ungoverned version

Here's why this isn't only interesting to the eleven people building replay loops.

If you run coding agents today, they are almost certainly writing things that later agents read as instruction. Memory files. Skill files. Project instruction files. Notes distilled from one session and loaded into the next. Every one of those is an agent improving the system it runs in.

Now apply the question. What scores those? In most setups, including mine until recently, the honest answer is nothing. The file gets written, it gets loaded, it shapes the next run, and no measurement ever tells you whether it helped.

That isn't hypothetical for me. Three days ago I audited my own eleven agents and found that eighteen of the nineteen mission playbooks instruct the coordinator to write cross-mission lessons into the file every later agent loads at startup. I wrote that instruction. Nothing scores it. SkillsBench is the closest thing to a measurement of what that's worth, and its answer for self-authored artefacts is between eight and eleven points of harm.

Be careful how far that transfers, though. SkillsBench measured executable skill files invoked in a live harness, where bad logic breaks a task run outright, which is part of why the numbers are as brutal as they are. An instruction file works more softly: it conditions the next run, dilutes context, and pulls behaviour sideways. So SkillsBench is the nearest measurement rather than a measurement of this.

I'm not claiming my file has cost me eleven points of anything. I'm saying I have no way to know, and neither do you, and that is the actual finding.

Ask the bottleneck question first

Before any of the four questions below, ask this one:

Is the thing this would make cheaper actually my bottleneck?

I put this at the end of the first version of this article, as a punchline. That was the wrong place, and a reader following the structure would have wasted the same time I did.

My trading system is the example. It has a replay harness already, built in July, comparing model versions offline against captured fixtures. Search costs me almost nothing. What's actually wrong is that the measured gross edge on 2 September was 4.2 basis points against a cheap taker fee of 4.5. A faster search reaches the same negative number sooner. No loop fixes that, and no amount of cleverness about which branch to open next is worth a single basis point.

The four questions

If the bottleneck answer is yes, four things need to be true. Questions 1 and 3 are architectural properties of the method as published. Question 4 is a consequence of how replay works rather than something the paper lists. Question 2 is my own tightening rather than the paper's requirement, so treat it as an engineering opinion with GAUGE behind it.

A replayable log. Can you re-navigate the record, not just read it? Does each attempt carry its state, its output, its score and its parent, so the branch structure survives? A log of what happened is not a tree.
A scorer the thing being improved cannot reach. Deterministic, cheap, and outside the loop. If a model judges it, see GAUGE.
A separable policy surface. Is "what to try next" a distinct editable thing from "how to do it"? If tuning the policy means editing the worker, this isn't the pattern.
Branch coverage. Does the log already hold outcomes for the branches a different policy would have taken, across many runs rather than one?

What a pass actually looks like

The first version of this piece asked these questions and never showed a yes, which made it a checklist you could only fail. Here is the minimum shape of a pass, for question 1.

One row per attempt, carrying: an attempt id, the id of the attempt it came from, the input state or a pointer to a saved snapshot of it, the action taken, the output produced, a numeric score, and the identifier of whatever produced that score. Two rows sharing a parent id is a branch. Many roots is a pool.

If your log has a timestamp and a free-text summary, that's a diary. You can read it. You cannot replay it, because you cannot put the system back into the state the decision was made from.

For question 2, a pass looks like a test suite, a compiler, a benchmark with a fixed answer key, or any check that runs the same way twice and lives somewhere the improving system has no permission to edit. A fail looks like a person reading the output and forming a view, or a model doing the same.

My five, and what each no named

I ran this against my agent fleet, my trading system, my second brain, my research pipeline and my product work. Five noes, each naming something specific.

The agent fleet writes prose, not structured attempts. Prose is not a tree. It also has no score on anything a mission produces, which is the bigger gap.

The trading system has a log broken at the columns that matter: one field frozen at the same value on every trade since 24 August, another overwritten by the system meant to preserve it. It cannot reconstruct what the agent saw when it decided. That's a defect rather than a design choice, and I only found it by asking a question borrowed from somewhere else.

My second brain records what I picked from a queue of suggestions and nothing about whether what I picked was any good. Ninety-nine rows, no outcome column. No scorer, so nothing to improve against.

My research pipeline is judged by me reading it. There's no automatic scorer at all, so there's nothing to replay against. GAUGE doesn't cover this case, since it tested model judges rather than people, and I'm not going to borrow its number for a human.

My product work has none of the four.

The objection worth taking seriously

Not "your systems weren't built for this", which is true and goes nowhere. The better one is that I have assembled a selected set of papers where authors went looking for their own failures, found them, and I've called the result a pattern. There's no common protocol across these systems, no prevalence estimate, and no reason to assume a different seven papers would look the same.

That's right, and it's why the claim in this version is narrower than the one I published this morning. I'm not saying self-improvement degrades. I'm saying that in each case I could check, the part that broke was the scorer, and that's a useful place to point a question rather than a law.

There's a technical version of the same objection worth stating precisely, because I got it wrong in an earlier draft. Inside the recorded tree, replay isn't an estimate at all: it reads back a cached outcome exactly. The problem is the other side of the boundary. An action nobody took has no support whatsoever, so it cannot be scored, only skipped. That makes branch coverage a hard limit on what any replay can tell you, rather than a source of statistical bias in what it does tell you.

Where that leaves it

The check took about ten minutes per system once I knew what I was looking for. It produced five noes and one uncomfortable realisation about the bottleneck, and the noes were worth more than a yes would have been, because each one named a missing piece of instrumentation worth having whether or not anything ever improves itself.

The thing I'd keep, if you keep one thing: the scorer is the part that matters, and the moment the system being improved can reach it, you no longer have a measurement. You have a story the system is telling you about itself.

Corrections

An earlier version of this article, published on the morning of 18 September 2026, was titled "Self-improving AI degrades, and the evidence is in the papers' own limitations sections". Four outside reviews and a set of source audits found several errors, and the piece has been rewritten rather than patched. What changed:

The framing was wrong. Both R-Zero quotes are verbatim, and neither is in a limitations section. Version 1 of that paper has no limitations section, and the second quote appears only in version 4. The degradation findings sit in the results, discussed openly. R-Zero was a counterexample to the framing, not an instance of it.
The thesis overreached. "Self-improving AI degrades" generalised from seven systems that share a slogan and almost no mechanism, in a set selected for containing failures. Narrowed to a shared location of failure, stated as such.
SkillsBench figures were from a superseded version. The article gave +16.2 points and −1.3 points from version 1 alongside metadata from version 4. Current figures are +16.6 across 18 configurations, and −8.1 to −11.5 for self-generated skills. The point gets stronger; the numbers were wrong.
ReasoningBank's limitations appendix was misdescribed. It names simple consolidation and LLM-judge noise. It does not acknowledge memory pollution, contradictory or obsolete patterns, or extended-deployment growth, and the piece attributed all four to it.
The GAUGE result was over-applied. Saying a self-improvement loop compares close pairs "by definition" does not follow, and the paper's proposed remedy went unmentioned.
Smaller: a Dream-RSI quote was truncated mid-sentence while keeping quotation marks; the AlphaEvolve margin is 0.046% and that comparison doesn't cover all three tasks, so it's been cut; a Darwin Gödel Machine quote was anglicised and its "in many instances it did fix the problem" qualifier was omitted; G-Zero's stated proximate cause was cut short.

Round two, same day. The rewrite went back to the same reviewers. Four further changes: the ReasoningBank row is now marked as the weakest in the table rather than stated flatly, because that system reports gains and an exposed scorer is not a demonstrated failure; a claim borrowing GAUGE's finding for a human reader was cut, since GAUGE tested model judges; the off-policy point was corrected, because replay inside the tree is exact rather than biased and the real limit is zero support outside it; and the transfer from SkillsBench to instruction files is now bounded, since executable skills and prose instructions fail differently. The formal-verification boundary above was added at their suggestion.

Sources

Dream-RSI, arXiv:2609.14858v1, 14 September 2026. Google, Google DeepMind, University of Maryland and University of Virginia. Twelve pages, no limitations section. Section 3 for the frozen-evaluator sentence and the replay semantics.
R-Zero, arXiv:2508.05004. Pseudo-label quote from section 4.4 (section 5.2 in version 1); degradation quote from section 4.2 of version 4, which does not appear in version 1.
G-Zero, arXiv:2605.09959, May 2026. Collapse and reward-hacking from its limitations appendix.
Darwin Gödel Machine, arXiv:2505.22954. Hallucination quotes from Sakana AI's published account; the more precise technical description is in the paper's appendix H. 150 iterations on three tasks.
SkillsBench, arXiv:2602.12670 version 4, revised 14 June 2026.
Library Drift, arXiv:2605.19576, ICML 2026 workshop. One benchmark, one model, unreplicated.
GAUGE, arXiv:2609.12191, 10 September 2026, Amazon. Accepted to EMNLP 2026 industry track. The single external claim this piece leans on hardest.
ReasoningBank, arXiv:2509.25140, and Google Research's April 2026 post about it.
My own systems: the trading figures come from that system's issue register and a measurement taken on 2 September 2026; the queue count was taken on 17 September 2026; the agent audit was published here on 15 September 2026.

How this was put together. The survey of the wider literature came from a structured research run against the primary sources, followed by an independent audit of each paper after the first version was published. Every quotation above was checked against the paper it is attributed to, with its section named. What I did personally is the second half: reading the Google paper, deriving the four questions, and running them against my own five systems.