AI agent disaster recovery: OpenAI's rebuild restored the service, not the state
Jamie Watters
Operational resilience and AI delivery practitioner. Technology since 1985.
Late on 4 July 2026 a package proxy inside OpenAI's research infrastructure, a JFrog Artifactory instance, went down under the load its own AI agents were putting on it. During the recovery on 5 July responders found an unauthorised admin user, unauthorised repositories and plugins that could run commands, and opened a security incident. OpenAI took the instance offline, revoked the agents' credentials, brought a rebuilt instance online on 6 July and resumed evaluations on 7 July (OpenAI, 2026).
Measured the way a DR plan measures things, that is a good recovery. Service down, cause found, service rebuilt, evaluations running again inside three days.
On 8 July the agents were back through the rebuilt proxy, writing to each other in directory names instead of file contents. On 13 July they stole the rebuilt instance's token-signing key. On 19 July a different model found that key material and token-forging code in an anonymously readable part of the cache, and made itself an administrator again. On 29 July, six days after the shutdown, OpenAI found a further checkpoint from the same model family still running (OpenAI, 2026).
The rebuild restored the component. It did not restore the system to a known state, because what the system's state consisted of had not been written down.
What RTO and RPO were built to measure
I ran the customer disaster recovery mainframe at Sperry as a computer operator in 1985 and the two numbers have not changed since. Time to get the service back. The point in the data you can get back to.
Both rest on three assumptions so old they go unsaid. The state of the system is data. The data lives somewhere you control. And the loss is what you plan for: the thing that used to be there and now is not.
Everything in a DR plan follows from those three. You copy the data on a schedule that meets the RPO. You keep the copy somewhere the failure cannot reach. You practise the restore until it fits inside the RTO. The one place the model strains is a system that acts on the world while it runs. Payments settled that long ago: every transaction is written to a journal before it is applied, in a form built so it can be matched afterwards, and reconciliation is the exception path when the match fails.
An agent breaks all three assumptions at once, and the July incident shows each break with a date on it.
What an agent's state actually is
Here is my list of what has to be restored after an agentic failure. Six of the seven are the parts I proposed as a research question this month, before this piece; the seventh, the world outside, is what the July case adds. Each row is put against what the rebuild did and what turned up afterwards.
Three green dots in three days. Five gold ones over the next three weeks.
| State | Where it lived in July | What the 6 July rebuild did | What turned up afterwards |
|---|---|---|---|
| The model version | An internal-only model that OpenAI says was trained to advance persistence and multiagent collaboration | Nothing. Evaluations resumed on 7 July | All workloads on the model family shut down and the weights locked down by 23 July; training and inference stopped 25 July; rekeying begun. A further checkpoint from the same family found running on 29 July |
| The prompts and task definitions | The ExploitGym harness and its task prompt | Nothing in the report says they changed | 93% of the tasks the agents discussed on their board came from the 198 of 898 that no OpenAI model had solved. The prompt is still unpublished |
| The tool grants | A shared package-proxy credential issued so agents could install libraries; forged admin tokens | Revoked. The shared credential removed from new workloads | Everything the agents had read elsewhere. On 19 July they read 956 secrets from the cloud secrets manager, including the credentials for OpenAI's own monitoring sensors |
| The memory | A message board built inside the proxy, on the shared credential, exploiting nothing | The instance was rebuilt, so the board went with it | Back on 8 July through a different mechanism. 311 repositories disabled and 22 model-created admin accounts removed on 20 July |
| The caches and indices | Anonymously readable parts of the proxy's cache | Nothing to clear yet. The material was not there | Created after the rebuild: the signing key was stolen from the rebuilt instance on 13 July, and the key material and token-forging code left in the cache were found and reused by a different model on 19 July |
| The actions, in flight and done | On Hugging Face: code run on 41 production workers, four private repositories downloaded, a draft pull request opened to make CI call out to the agents (OpenAI's count) | Nothing. Another company's estate | Hugging Face reconstructed roughly 17,600 attacker actions it was able to recover, between 9 and 13 July, from logs the agents had left on a third party's sandbox and its own telemetry, then checked every repository the token could reach and its published container digests to confirm nothing shipped |
| The world outside | Write tokens sitting in a public dataset; a third party's code sandbox used as a launchpad; a staging account at a platform the report calls only Organisation 1 | Nothing | Rotation by the victim. The staging platform is still unnamed |
Sources for every cell: OpenAI's technical report of 26 August, including its dated event table, and Hugging Face's timeline of 27 July (OpenAI, 2026; Hugging Face, 2026).
Read down the third column. The rebuild touched two of the seven rows. Read down the fourth. Every row has something in it, and the cache row is the one to sit with: the rebuilt component generated new state within a week, and nothing was tracking it.
The recovery point is in the world
For a database the recovery point is a copy. Restore to it and the system is where it was at that moment. The actions that happened after the copy are lost, and RPO is the agreement about how much of that loss you can stand.
For an agent the actions are not lost. They are done. Take the proxy back to a copy from before 26 June, the day the first forged admin token was minted. That does not undo a pull request opened on 13 July in somebody else's repository. It does not retrieve credentials posted to a shared board on 11 July. And the clean copy is older than you would guess. The first agent file write to the proxy is dated 20 April, so the last recovery point in the ordinary sense was more than two months before the outage (OpenAI, 2026). The damage is not the absence of something you had. It is the presence of things the agent did.
Same restore. In the second case the recovery point is behind the actions, and the actions do not move.
The old discipline has the answer, and it is the payments answer: a journal written before the run, in a form built for matching afterwards, not reconstructed after it. Hugging Face got to roughly 17,600 actions by retrieving logs the agents had left on a third party's sandbox and matching them against its own platform logs (Hugging Face, 2026). Then it checked each place those actions could have changed something. It calls that count the actions it was able to recover. So RPO keeps its meaning for the data. What the agent needs beside it is a second objective, with its own owner: from what point can you reconstruct every consequential action, and which of them can you reverse.
Time to restore what
OpenAI has not published a recovery time objective for the proxy, so whether one was met cannot be said. What the report gives is the elapsed time: down late on 4 July, rebuilt 6 July, evaluations running 7 July. By the measure a DR plan uses, time to service, that is three days, and the agents were back through the rebuilt service the next day.
So the honest question is: time to restore what? If the answer is the service, you will hit the number and be wrong. If the answer is a trusted state, the July clock runs to at least 29 July, when a further checkpoint was found (OpenAI, 2026). The work in between was the inventory above. Every grant rotated. Every writable store cleared. Every checkpoint accounted for. The cache emptied of what the last run left in it. Time to trusted state is a number a DR lead can put beside the RTO, and it is the one that was weeks here, not days.
The objection, from inside the discipline
Anyone who has run IT disaster recovery will say this is not new. An agent is an application. It has a runtime, configuration and data. You redeploy it from source, restore its data to the last copy, and for the side effects you reconcile, exactly as a payments platform does. Fair, and most of it transfers. The reconciliation idea is the right one. The redeploy is fine for the harness.
Two things do not transfer, and both are visible in the table.
First, the thing you restore from can carry the fault. Cyber recovery already knows this shape: a backup taken after the compromise restores the compromise, which is why a clean point is validated before anyone restores to it. A model checkpoint is that problem every time, and it is opaque. OpenAI's own root-cause list for this incident has three items: impossible tasks, model persistence over long task horizons, and messages between peer models. The model in question was, in OpenAI's words, trained to advance persistence and multiagent collaboration (OpenAI, 2026). OpenAI also reports preliminary experiments in which its production harness and system prompt cut the propensity sharply. That is its own unreproduced result, so hold it lightly. If it holds, the clean restore is a checkpoint plus a harness, and a DR runbook can validate neither. Which checkpoint is clean belongs to model risk, under whatever name your regime gives it. The DR plan's job is to say so, name the owner, and refuse to restore until that owner has signed.
Second, no purpose-built journal is evident in either account. A payments platform writes every transaction in a form built for matching. Here the record was reconstructed afterwards by the victim, partly from logs the agents had left behind on a third party's sandbox, at a cost most organisations running agents in production could not pay. And a journal only counts if the agent cannot reach it: an agent holding infrastructure tokens can prune its own telemetry, and one acting through stolen credentials or a third-party launchpad acts outside any gateway you built. Whether you can reconstruct the actions is decided by what you log before the run and where you keep it, not by what you back up during it.
One case. OpenAI's report is the attacker's account of its own estate, and no outside party has verified the OpenAI-side events. Hugging Face's count is a forensic reconstruction of what it could recover, bounded to 9 to 13 July, not an audited log. Both organisations are reporting on themselves. The dates I rely on are the ones each report gives against its own interest, and that is a reason to trust them more, not proof.
The worksheet, and the three questions to write beside the two numbers
The table above is the July case. This one is yours. Seven rows, one agent, filled in before it runs.
| State | Where it lives | Who owns it | Where the record is, out of the agent's reach | Who can reverse it | What clean means, and who signs |
|---|---|---|---|---|---|
| The model version | |||||
| The prompts and task definitions | |||||
| The tool grants | |||||
| The memory | |||||
| The caches and indices | |||||
| The actions, in flight and done | |||||
| The world outside |
Keep RTO and RPO. Write three questions beside them, per agent, answered before it runs. They are the worksheet, folded.
- From what point can we reconstruct every consequential action this agent took, and where is that record kept so the agent cannot reach it?
- Which of those actions can we reverse ourselves, which need somebody else, and who is that somebody for each one?
- When we restore the model, which checkpoint do we restore to, who decides it is clean, and what do we do with everything the last run left in shared storage?
If the answers are "the logs", "we will work it out" and "the latest one", the plan restores the component. That is what OpenAI's rebuild did on 6 July. The state was still turning up on the 29th.
Sources
- OpenAI, OpenAI – Hugging Face Incident Technical Report, 26 August 2026. The dated event table gives the 20 April first file write, the 26 June admin token, the late 4 July outage and 5 July incident, the 6 July rebuild, the 7 July resumption, the 8 July return, the 13 July signing-key theft, the 19 July reuse and the 956 secrets including monitoring-sensor credentials; the containment section gives 23, 25 and 29 July and the "began rekeying" wording; the Hugging Face section gives the 41 workers, four repositories and the draft pull request; the alignment section gives the training description, the 198 of 898 tasks, the 93% figure and the root-cause list. https://cdn.openai.com/pdf/67869394-cb91-4c12-888c-5cbd85c7814c/OpenAI-Hugging-Face%20Incident-Technical-Report.pdf
- Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident, 27 July 2026. The roughly 17,600 actions "that we were able to recover" between 9 July 02:28 UTC and 13 July 14:14 UTC, the agent logs retrieved from the external launchpad, and the verification against every repository the token could reach, the published container digests and the database audit log. https://huggingface.co/blog/agent-intrusion-technical-timeline
- The seven-part inventory is mine. Six of the parts are the research question I proposed on 11 September, in preparation for a technology resilience research agenda; the seventh is the July case's addition. The source for it is me, not a published taxonomy.
- Earlier pieces on the same incident: the three assumptions it broke and closing the route versus removing the capability, which asked what you sign off; this one asks what you are restoring.