Self-Play

Self-play is a reinforcement learning method in which a system generates its own training experience — originally by playing games against copies of itself, and more recently by having a language model write the tasks it then learns to solve. Its attraction is the same in both forms: the difficulty of the training data rises automatically with the learner's skill, and no human has to supply the examples. Its limitation is also the same: something outside the learner has to decide who won.

The Classic Form: Board Games

The original AlphaGo was first trained on thousands of human amateur and professional games. AlphaGo Zero (DeepMind, October 2017) removed that step: starting from random play and using only the board position as input, it beat the earlier version 100 games to 0 after three days of self-play. AlphaZero (Silver et al., December 2017) generalised the recipe, reaching superhuman play in chess, shogi and Go within 24 hours from the rules alone, with Monte Carlo tree search guided by a single network.

Three properties made this work, and they are worth stating because language-model self-play lacks at least one of them. The game has two symmetric players, so a copy of the agent is always a well-matched opponent. The rules are fixed and can be simulated exactly. And the outcome — win, lose, draw — is unambiguous and free to compute.

Self-Play for Language Models

Most useful language tasks have no opponent, so self-play is recast as two roles played by one model: a proposer that invents tasks and a solver that attempts them. Absolute Zero (Zhao et al., May 2025) established the pattern for reasoning: a single model proposes coding tasks chosen to maximise its own learning progress and solves them, with a code executor both validating the tasks and checking the answers. The authors report that the resulting model, trained on no external data, outperformed models trained on tens of thousands of curated in-domain examples on coding and maths benchmarks.

SPADE (Liu et al., August 2026, marked by its authors as work in progress) extends the proposer from writing problems to writing whole RL environments as executable code, covering multi-step tool use as well as reasoning. Its Environment Designer is rewarded by a regret signal — the gap between the agent's reward with and without privileged hints — which favours tasks that are solvable but not yet solved. At 30B parameters the paper reports +5.3 points on average over fixed-environment baselines across eight benchmarks, and larger gains on tool use (+5.7 on BFCL-v4, +13.9 on ACEBench-Agent). These are single-paper results from the method's authors.

System	What the model generates	What decides success
AlphaGo Zero / AlphaZero (2017)	Games against itself	Game rules
Absolute Zero (2025)	Coding and reasoning tasks	Code executor
SPADE (2026)	Executable environments	The generated environment's own checks, plus a regret signal
G-Zero (2026)	Challenging prompts and hints for open-ended generation	An intrinsic reward (shift in the model's own predictions when given a hint)
J-Zero (2026)	Tasks, answers and judgements	A co-trained Judge

Tasks Without a Ground-Truth Checker

The frontier question is whether self-play survives the loss of the referee. G-Zero (Huang et al., May 2026) targets open-ended generation, where the usual fallback — an LLM judge — invites reward gaming. It replaces the judge with an intrinsic signal: how much the generator's response shifts when it is given a self-generated hint. J-Zero (Chu et al., August 2026) instead trains the referee as a third role, co-evolving a Challenger, a Solver and a Judge; the authors report gains of about 4.2 points on verifiable problems and 8.0 on subjective domains over prior methods, and that their system kept improving for at least ten iterations where baselines degraded after two.

Both are early, single-paper results, and both replace an external fact with something the model produces. G-Zero's own theoretical guarantee is conditional on adequate exploration by the proposer and on filtering that keeps pseudo-label noise low. Whether such loops keep tracking real quality over long horizons, rather than the Judge and Solver drifting together, is not yet established.

Stability

Self-play loops fail in a characteristic way: the proposer finds tasks that the solver can score on without learning anything, the data narrows, and performance degrades — a relative of model collapse and of reward hacking. "Survive or Collapse" (Pu et al., May 2026) studied this in controlled code-prediction and domain-specific-language tasks and found an asymmetry: a strict filter on which generated examples enter training was sufficient for stability under every reward variant they tried, while no reward design alone prevented collapse once the filter was removed. They also report a "grounded proposer paradox", in which a proposer with access to ground truth accelerated collapse when paired with a self-consistency solver, and observe that training metrics diverged before validation accuracy visibly fell.

The practical reading, with the caveat that it rests on one study in narrow domains, is that the gate matters more than the reward — and that a healthy-looking accuracy curve is not evidence that a self-play loop is stable.