# Self-Play

> Self-play is a training method in which an AI system improves by generating its own opponents or tasks instead of learning from external data.

Source: https://metavert.io/self-play  
Published: 2026-10-07  
Updated: 2026-10-07

**Self-play** is a [reinforcement learning](https://metavert.io/reinforcement-learning) method in which a system generates its own training experience — originally by playing games against copies of itself, and more recently by having a language model write the tasks it then learns to solve. Its attraction is the same in both forms: the difficulty of the training data rises automatically with the learner's skill, and no human has to supply the examples. Its limitation is also the same: something outside the learner has to decide who won.

### The Classic Form: Board Games

The original [AlphaGo](https://metavert.io/alphago) was first trained on thousands of human amateur and professional games. AlphaGo Zero (DeepMind, October 2017) removed that step: starting from random play and using only the board position as input, it beat the earlier version 100 games to 0 after three days of self-play. [AlphaZero](https://metavert.io/alphazero) (Silver et al., December 2017) generalised the recipe, reaching superhuman play in chess, shogi and Go within 24 hours from the rules alone, with [Monte Carlo tree search](https://metavert.io/monte-carlo-tree-search) guided by a single network.

Three properties made this work, and they are worth stating because language-model self-play lacks at least one of them. The game has two symmetric players, so a copy of the agent is always a well-matched opponent. The rules are fixed and can be simulated exactly. And the outcome — win, lose, draw — is unambiguous and free to compute.

### Self-Play for Language Models

Most useful language tasks have no opponent, so self-play is recast as two roles played by one model: a *proposer* that invents tasks and a *solver* that attempts them. Absolute Zero (Zhao et al., May 2025) established the pattern for reasoning: a single model proposes coding tasks chosen to maximise its own learning progress and solves them, with a code executor both validating the tasks and checking the answers. The authors report that the resulting model, trained on no external data, outperformed models trained on tens of thousands of curated in-domain examples on coding and maths benchmarks.

SPADE (Liu et al., August 2026, marked by its authors as work in progress) extends the proposer from writing problems to writing whole [RL environments](https://metavert.io/rl-environments) as executable code, covering multi-step tool use as well as reasoning. Its Environment Designer is rewarded by a regret signal — the gap between the agent's reward with and without privileged hints — which favours tasks that are solvable but not yet solved. At 30B parameters the paper reports +5.3 points on average over fixed-environment baselines across eight benchmarks, and larger gains on tool use (+5.7 on BFCL-v4, +13.9 on ACEBench-Agent). These are single-paper results from the method's authors.

| System | What the model generates | What decides success |
| --- | --- | --- |
| AlphaGo Zero / AlphaZero (2017) | Games against itself | Game rules |
| Absolute Zero (2025) | Coding and reasoning tasks | Code executor |
| SPADE (2026) | Executable environments | The generated environment's own checks, plus a regret signal |
| G-Zero (2026) | Challenging prompts and hints for open-ended generation | An intrinsic reward (shift in the model's own predictions when given a hint) |
| J-Zero (2026) | Tasks, answers and judgements | A co-trained Judge |

### Tasks Without a Ground-Truth Checker

The frontier question is whether self-play survives the loss of the referee. G-Zero (Huang et al., May 2026) targets open-ended generation, where the usual fallback — an LLM judge — invites reward gaming. It replaces the judge with an intrinsic signal: how much the generator's response shifts when it is given a self-generated hint. J-Zero (Chu et al., August 2026) instead trains the referee as a third role, co-evolving a Challenger, a Solver and a Judge; the authors report gains of about 4.2 points on verifiable problems and 8.0 on subjective domains over prior methods, and that their system kept improving for at least ten iterations where baselines degraded after two.

Both are early, single-paper results, and both replace an external fact with something the model produces. G-Zero's own theoretical guarantee is conditional on adequate exploration by the proposer and on filtering that keeps pseudo-label noise low. Whether such loops keep tracking real quality over long horizons, rather than the Judge and Solver drifting together, is not yet established.

### Stability

Self-play loops fail in a characteristic way: the proposer finds tasks that the solver can score on without learning anything, the data narrows, and performance degrades — a relative of [model collapse](https://metavert.io/model-collapse) and of [reward hacking](https://metavert.io/reward-hacking). "Survive or Collapse" (Pu et al., May 2026) studied this in controlled code-prediction and domain-specific-language tasks and found an asymmetry: a strict filter on which generated examples enter training was sufficient for stability under every reward variant they tried, while no reward design alone prevented collapse once the filter was removed. They also report a "grounded proposer paradox", in which a proposer with access to ground truth accelerated collapse when paired with a self-consistency solver, and observe that training metrics diverged before validation accuracy visibly fell.

The practical reading, with the caveat that it rests on one study in narrow domains, is that the gate matters more than the reward — and that a healthy-looking accuracy curve is not evidence that a self-play loop is stable.

## Related Topics

- [Reinforcement Learning](https://metavert.io/reinforcement-learning) — The field self-play belongs to
- [AlphaZero](https://metavert.io/alphazero) — The canonical self-play system for board games
- [AlphaGo](https://metavert.io/alphago) — Where self-play first reached public attention
- [RL Environments](https://metavert.io/rl-environments) — What LLM self-play increasingly generates
- [Model Collapse](https://metavert.io/model-collapse) — The degradation risk in any self-generated data loop
- [Reward Hacking](https://metavert.io/reward-hacking) — How proposer and solver can game each other
- [GRPO](https://metavert.io/grpo) — The policy-gradient algorithm most of these systems train with
- [Synthetic Data](https://metavert.io/synthetic-data) — Self-generated tasks as a form of synthetic data

## Further Reading

- [AlphaGo Zero: Starting from scratch](https://deepmind.google/discover/blog/alphago-zero-starting-from-scratch/) — Google DeepMind, October 2017
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://arxiv.org/abs/1712.01815) — Silver et al., arXiv, December 2017
- [Absolute Zero: Reinforced Self-play Reasoning with Zero Data](https://arxiv.org/abs/2505.03335) — Zhao et al., arXiv, May 2025
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://arxiv.org/abs/2608.19197) — Liu et al., arXiv, August 2026
- [G-Zero: Self-Play for Open-Ended Generation from Zero Data](https://arxiv.org/abs/2605.09959) — Huang et al., arXiv, May 2026
- [J-Zero: Unified Challenger-Solver-Judge Self-Evolution from Zero Data](https://arxiv.org/abs/2608.26582) — Chu et al., arXiv, August 2026
- [Survive or Collapse: The Asymmetric Roles of Data Gating and Reward Grounding in Self-Play RL](https://arxiv.org/abs/2605.22217) — Pu et al., arXiv, May 2026