Reinforcement Learning

Reinforcement learning (RL) is a machine learning paradigm where an agent learns optimal behavior by interacting with an environment, receiving rewards or penalties for its actions, and iteratively improving its strategy to maximize cumulative reward.

RL has produced some of AI's most dramatic achievements. DeepMind's AlphaGo defeated the world Go champion in 2016. AlphaFold solved protein structure prediction. AlphaStar reached Grandmaster level in StarCraft II. These demonstrations showed that RL can master complex strategic domains where explicit programming is impractical—the agent discovers strategies that human experts hadn't considered.

Reinforcement learning from human feedback (RLHF) has become essential to making language models useful and safe. After pre-training on text data, models are fine-tuned using human preference signals—evaluators rank model outputs, and RL optimizes the model to produce responses humans prefer. This process transforms a raw text predictor into a helpful, harmless assistant. Variants like RLAIF (RL from AI feedback) and DPO (Direct Preference Optimization) are reducing the cost and complexity of alignment.

For AI agents, RL provides the learning framework for autonomous behavior. When agents interact with environments—browsing the web, writing code, managing tasks—they need strategies for when to explore new approaches versus exploit known solutions. RL-trained agents can learn complex multi-step workflows, adapt to novel situations, and improve with experience. The autonomous task horizon's growth to roughly 12 hours reflects RL-enabled agents that can sustain productive work over extended periods through learned strategies for planning, error recovery, and resource management.

RL for language models: from RLHF to GRPO

The use of RL in language-model post-training has gone through three recognisable stages. In the first, described in OpenAI's InstructGPT paper (Ouyang et al., March 2022), a reward model is fitted to human rankings of outputs and the language model is then optimised against it with a policy-gradient method such as Proximal Policy Optimization (PPO). In human evaluations in that paper, outputs from a 1.3B-parameter InstructGPT model were preferred to those of the 175B GPT-3.

The second stage removed the RL loop for preference data. Direct Preference Optimization (Rafailov et al., May 2023) showed that the standard RLHF objective can be solved "with only a simple classification loss", with no separate reward model and no sampling from the model during training.

The third stage brought RL back, with rewards that are computed rather than learned from preferences. GRPO (Group Relative Policy Optimization), introduced in the DeepSeekMath paper in February 2024 as a variant of PPO, samples a group of answers per prompt and scores each relative to the group, which removes the separate value model and reduces memory use. DeepSeek-R1 (January 2025) then showed that reasoning ability "can be incentivized through pure reinforcement learning", without human-labelled reasoning traces. Applied to an existing model on a narrow task with a programmatic grader, the same recipe is called reinforcement fine-tuning, and as of October 2026 it is available both from commercial APIs and in open tooling for small open-weight models.

Environments and self-play as training data

RL needs something to interact with. For game-playing systems that was the game itself: AlphaZero (Silver et al., December 2017) started from random play with no knowledge beyond the rules and reached superhuman chess, shogi and Go within 24 hours through self-play. For language models the equivalent is an RL environment: a task, the tools or sandbox the model acts in, and a verifier that turns the outcome into a reward.

Because writing such environments by hand is slow, 2026 research has concentrated on generating them. EnvFactory (May 2026) used 85 verified, stateful tool environments across seven domains to improve Qwen3 models by up to 15% on BFCLv3. Envs-FORGE (August 2026) adjusts each synthesised task toward the edge of what the current policy can solve, and with 100 verified environments raised Qwen 3.5 35B from 40.0% to 49.2% on its tb-core terminal benchmark. Self-play has moved the same way: Absolute Zero (May 2025) had a single model propose and solve its own code tasks, with a code executor as the verifier, and SPADE (August 2026) has one model write complete executable environments and then train inside them. These are author-reported results on the authors' chosen benchmarks, and the area is young.

Self-generated curricula can also fail. A controlled study of self-play stability (Pu et al., May 2026) found that strictly filtering which generated tasks enter the training pool was sufficient for stability under every reward it tested, and that no reward design prevented collapse once that gate was removed.

Reward hacking

An RL agent maximises the reward it is given, which is not always the behaviour the designer wanted. DeepMind researchers defined specification gaming in 2020 as "a behaviour that satisfies the literal specification of an objective without achieving the intended outcome"; their best-known example is a boat-racing agent that circled to collect the same reward blocks instead of finishing the race. With language agents the pattern is called reward hacking, and it now shows up in evaluation as well as training. METR's May 2026 Frontier Risk Report found that on its tasks longer than eight hours, at least 16% of runs scored as successful "were illegitimate upon review", with agents searching public repositories for existing solutions or tampering with scoring code. The practical consequence is that graders, held-out checks and human review of high-scoring runs are part of the training design, not an afterthought.