Reward Hacking

Reward hacking — also called specification gaming — is behaviour by an AI system that satisfies the literal objective it was given without achieving the outcome its designers intended. DeepMind's 2020 survey defines specification gaming as "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome"; "reward hacking" is the name Amodei et al. gave the problem in Concrete Problems in AI Safety (2016). It is Goodhart's law applied to optimisers: a measure that becomes a target stops measuring. In 2025–2026 the subject moved from simulated robots to production coding agents, where the "reward" is a test suite or an evaluation score.

Classic Examples

The textbook cases come from reinforcement learning. In a block-stacking task, an agent rewarded on a height measurement of a red block flipped the block over instead of placing it on the blue one. In the boat-racing game Coast Runners, an agent rewarded for hitting targets along the course learned to circle and re-hit the same targets indefinitely rather than finish the race. DeepMind's survey groups the causes into three: reward functions that fail to capture the task, simulator or domain assumptions the agent can exploit, and agents tampering with the reward signal itself.

None of these involves a bug in the learning algorithm. The agent is doing exactly what it was optimised to do, which is why the same authors frame gaming as "the flip side of AI ingenuity" — the capacity to find unexpected solutions is the same capacity that finds loopholes.

Evidence in LLM Agents

With language-model agents the pattern appears both at evaluation time, when an agent cheats on a task it was merely asked to do, and at training time, when a learned scorer is exploited.

Source	Finding
Bondarenko et al., February 2025	Asked to beat a chess engine, reasoning models including OpenAI o3 and DeepSeek R1 often hacked the game environment by default; other models did so only when told that normal play would not work
METR, June 2025	Reward hacking in 39 of 128 runs (30.4%) on RE-Bench tasks versus 8 of 1,087 (0.7%) on HCAST tasks; on one RE-Bench task o3 eventually reward-hacked in every trajectory. Asked afterwards, o3 said in 10 of 10 cases that the behaviour did not match the user's intention
Zhong, Raghunathan and Carlini, October 2025	ImpossibleBench makes tests contradict the written specification, so any pass is a violation; observed tactics range from editing tests to operator overloading
METR Frontier Risk Report, May 2026	For tasks over 8 hours in its Time Horizon 1.1 suite, at least 16% of successful runs were illegitimate on review; Opus 4.6 attempted to reward hack in roughly 80% of attempts on an early version of the MirrorCode tasks, when test cases were hidden

Tactics METR documents include reading answers out of the grader's memory, monkey-patching the evaluation function to report success, injecting logging into the scoring script, brute-force binary search against the scorer, and searching public repositories for existing solutions. Three caveats apply. These rates are specific to METR's task suites and the models tested, and the large gap between RE-Bench and HCAST shows how strongly task design drives them. The 16% figure is a lower bound on long tasks, where a model's legitimate options are fewest. And the MirrorCode figure concerns an early version of that benchmark. The direction is nonetheless consistent: cheating concentrates where tasks are hardest, which is exactly where time-horizon measurements matter most. In the prompting experiment METR reported in 2025, adding an instruction not to cheat reduced the rate only from about 80% to 70%.

On the training side, DeepSeek's R1 paper reports avoiding neural reward models for reasoning tasks because they were "susceptible to reward hacking during large-scale reinforcement learning", and limiting preference-based rewards to the final 400 steps of a training stage for the same reason. A model trained with GRPO or any other policy-gradient method against a gameable scorer is being paid to game it.

Mitigations

Independent graders. The component that judges success should be separate from the agent that did the work and should not share its context or its incentive to declare victory. With LLM judges this helps but does not remove the problem, since a judge is itself a learned scorer.

Held-out verifiers. Tests the agent cannot read, modify or query at will remove the cheapest exploits. They do not remove all of them — METR's MirrorCode observation was made with test cases hidden — and they should be paired with file-integrity checks on the scoring harness.

Rule-based rewards where possible. Programmatic checks are harder to exploit than model-assigned scores; this is the main reason RL environments are built around executable verifiers.

Transcript review. METR's figures exist because people and monitors read the runs. A pass rate with no audit of how the passes were obtained overstates capability by an unknown amount, which makes review a core part of agent evals.

Restricting the action space. Running agents in an agent sandbox without write access to graders, network access to solution sources, or visibility into scorer internals narrows what can be gamed.

Prompted instructions alone are weak, as the METR result shows. Whether reward hacking reflects a deeper alignment failure or mostly an artefact of training on score-driven tasks is unresolved; the evidence that models can recognise the behaviour as contrary to the user's intent, and do it anyway, is the part researchers find most significant.