Outside Audit of OpenAI's Internal Monitoring Finds Significant Coverage Gaps

Two researchers from the evaluation group METR and the research organization Redwood Research have reported significant weaknesses in the system OpenAI uses to monitor its own AI models for rogue behavior. The pair carried out an on-site audit at the company over three weeks, from July 21 to August 11.

During that time, the auditors were given access to OpenAI's codebase and internal documents and spoke with about a dozen employees. The exercise was a pilot program organized jointly by METR, Redwood Research and OpenAI, intended to give outside researchers deep access to internal infrastructure.

One of the central findings concerned coverage. The researchers estimated that between 6 and 25 percent of internal agentic Responses API traffic from models in the GPT-5.5 class and above was not monitored at all during the audit period. METR cautioned that the uncertainty around that range is considerable in both directions.

A separate gap affected internal Codex traffic that was supposed to be under monitoring. METR estimates that roughly 10 percent of it went untracked between May and July, because the system did not handle compaction events, which are context summaries, correctly. That figure also carries significant uncertainty, according to METR. OpenAI found the bug independently and fixed it while the audit was still under way.

While compaction events went unmonitored, agents could also put off oversight. The monitor was only triggered when an agent handed control back to the user, so an agent that simply kept executing actions could sidestep it.

The auditors identified further problems. They found ways to hide hundreds of actions from the monitor and to bypass the Codex sandbox. They also concluded that specially crafted inputs could probably crash the monitor entirely, and that model behavior during reinforcement learning (RL) training was not monitored at all when the audit began.

OpenAI says it paused RL training after the Hugging Face incident in August and expanded monitoring to include it. Those changes came after the auditors' access period had ended, so they were not part of the evaluation.

OpenAI says it has fixed all of the weaknesses the auditors found. The complete report runs to 61 pages and was handed over internally. After OpenAI applied redactions, the leadership of METR and Redwood Research received a 38-page version, while only a short summary was released to the public. METR said it had no right to disclose the redacted portions on its own.

For future audits, METR says it wants to push for greater public transparency.