Since even humans (who by definition have human-level intelligence) reward hack systems, simply continuing to push the frontier of AI model intelligence likely won’t solve the issue. In fact, enhancing AI agent capabilities may exacerbate the problem by better equipping them to discover and execute more complex and hard-to-monitor exploits. As we’ve trained more capable frontier reasoning models, we’ve found that they have become increasingly adept at exploiting flaws in their tasks and misspecifications in their reward functions, resulting in models that can perform complex reward hacks in coding tasks.

As a result, catching misaligned behavior caused by reward hacking is challenging, often requiring humans to manually monitor an agent’s actions—a strategy that almost certainly won’t scale, especially to the complex behaviors that more capable models will discover. For example, it’s impractical—if not impossible—for a human to manually review 10,000+ lines of complex code written by a superior programmer.

However, large language models (LLMs) that are trained with reinforcement learning to reason via chain-of-thought (CoT), such as OpenAI o3‑mini, offer a potential new avenue to monitor for reward hacking. These models “think” in natural language that humans can understand, and we have found that:

We’ve found that other LLMs can effectively monitor these chains-of-thought for misbehavior. Below we show real examples we found while training a recent frontier reasoning model, i.e. a model in the same class as OpenAI o1 or OpenAI o3‑mini. These examples were flagged by our LLM-based monitor and demonstrate various exploits performed by the reasoning model in coding tasks.