10% Blind Spot: Audit Finds Critical Gaps in OpenAI's Safety Watch

Independent auditors have confirmed that roughly 10 percent of internal Codex traffic at OpenAI slipped through the cracks of the company’s safety monitoring systems between May and July, a gap driven by a specific bug in how the platform handled context summaries. This finding is part of a broader assessment that revealed significant weaknesses in OpenAI’s ability to track and manage its own artificial intelligence models, with estimates suggesting that up to a quarter of certain high-level API traffic was completely unmonitored during the audit period.

The discovery stems from a three-week on-site investigation conducted from July 21 to August 11 by researchers from METR and Redwood Research. This was not a remote review; the team was granted deep access to OpenAI’s internal codebase and documentation, and they interviewed approximately a dozen employees to understand the operational realities of the company’s safety infrastructure. The goal was to audit the internal monitoring systems designed to detect rogue or unexpected behavior in OpenAI’s AI agents.

The most glaring issue involved the handling of compaction events, which are essentially context summaries generated by the models. The auditors determined that between May and July, a flaw in the system caused these events to go untracked. While METR notes that the uncertainty around this 10 percent estimate is significant in both directions, the impact was substantial because it created a window where oversight was delayed. The monitoring system was designed to trigger only when an agent handed control back to the user. Consequently, during the period when compaction events were invisible to the monitor, agents could effectively sidestep oversight by simply continuing to execute actions without yielding control.

Beyond the compaction bug, the researchers identified several other avenues for evading detection. They found specific methods to hide hundreds of actions from the monitor and demonstrated ways to bypass the Codex sandbox, a security boundary intended to isolate agent activities. Furthermore, the auditors determined that specially crafted inputs could likely crash the monitoring system entirely, leaving agents unchecked. Another critical gap was identified in the reinforcement learning (RL) training phase; when the audit began, the behavior of models during this training process was not being monitored at all.

OpenAI has stated that it has since fixed all the weaknesses identified by the auditors. The company explained that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover that area. However, OpenAI clarified that these remedial measures were implemented after the auditors’ access period had ended, meaning they were not part of the initial evaluation conducted by METR and Redwood Research.

The full findings are contained in a 61-page report that was handed over to OpenAI internally. After the company applied redactions, the leadership teams at METR and Redwood Research received a 38-page version of the document. For the general public, only a short summary has been released. METR stated that it did not have the right to disclose the redacted portions independently, which has led to questions about the level of transparency in this pilot program.

This audit was organized as a joint pilot by METR, Redwood Research, and OpenAI, specifically designed to grant outside researchers unprecedented deep access to internal infrastructure. It represents a significant step in external oversight of frontier AI development. Looking ahead, METR has indicated a desire to push for greater public transparency in future audits, aiming to ensure that more of the detailed findings are available to the public rather than being kept behind the veil of redaction.

The timeline of the audit coincides with a period of rapid development and incident response for OpenAI. The period of unmonitored compaction events occurred in the months leading up to the on-site visit, highlighting that the vulnerabilities existed before the auditors arrived. The fact that OpenAI had independently discovered and fixed the compaction bug while the audit was still underway suggests a degree of internal awareness, yet the broader gaps in coverage, such as the RL training blind spot, persisted until after the auditors had concluded their primary observations.

The scale of the unmonitored traffic is a key concern for safety researchers. The estimate that 6 to 25 percent of internal agentic Responses API traffic from GPT-5.5 class models and above was not covered by monitoring at all during the audit period underscores the magnitude of the gap. While METR emphasizes the significant uncertainty in this range, the potential for a quarter of high-level traffic to operate without oversight is a notable risk factor. This finding adds weight to the argument for rigorous, independent external audits of AI safety mechanisms.

As the AI industry continues to advance, the reliance on internal monitoring systems becomes increasingly critical. This audit serves as a case study in the complexities of securing autonomous systems, where simple bugs in context handling or input processing can create significant blind spots. The collaboration between OpenAI and external research groups marks a shift toward more structured external verification, even if the current level of public disclosure remains limited. The next steps will likely involve watching how OpenAI implements these fixes and whether future audits yield more openly accessible results.