Recursive Language Models vs Retrieval-Augmented Generation

Selecting evidence before reading versus while reading

RAG, introduced by Lewis et al. in May 2020, commits to its evidence before the generator sees anything. That is its efficiency and its weakness. If the retriever surfaces the right passages, the model reads a few thousand tokens instead of millions. If the question needs evidence the retriever did not rank highly, or needs nearly all of the corpus, the generator never sees what it needs. Related techniques such as semantic search over embeddings and GraphRAG improve the selection step but keep the same shape.

The RLM paper (Zhang, Kraska and Khattab; v3, May 2026) moves selection inside the loop. The model can grep the input, sample it, partition it and hand each partition to a fresh model call, then combine the returned values in code. Retrieval becomes one strategy the model may choose rather than a fixed stage, which is why the two approaches are better seen as overlapping than as opposites: an RLM can run a keyword search, and an agentic RAG system can issue several queries.

What the benchmarks show, and their limits

On BrowseComp-Plus with 1,000 documents per task (6M–11M tokens), the RLM authors report 91.3% for an RLM built on GPT-5 against 51.0% for a CodeAct agent equipped with BM25 retrieval. On OOLONG, a task that requires processing most of a 131K-token input, their table gives 56.0 for the RLM and 38.0 for the retrieval agent. Both gaps favour the RLM, and the OOLONG gap fits the mechanism: aggregation over every entry is the case top-k retrieval handles worst.

Three caveats apply. The retrieval baseline is a single lexical (BM25) configuration inside a coding agent, not a tuned dense-retrieval and reranking pipeline, so the result does not establish that RLMs beat RAG in general. The comparison comes from the paper proposing the method. And a March 2026 preprint from Apple researchers (Alizadeh et al.) argues that "recursion itself is not the primary driver of performance": a self-reflective program search without sub-calls matched or exceeded RLMs in their tests, and RLMs were weaker on semantically intensive tasks. No study opened for this page compares an RLM with a modern production RAG stack.

Cost, latency and operations

RAG's costs are front-loaded: building and refreshing an index, then a cheap query path. Li et al. (July 2024) found that long-context models beat RAG on average quality but that RAG's "significantly lower cost remains a distinct advantage". RLM costs are back-loaded and variable. The authors report an average of $0.99 per BrowseComp-Plus query with a standard deviation larger than the mean, describe median runs as cheap and outlier trajectories as expensive, and name runaway sub-call costs as an open problem.

Operationally, RAG needs retrieval infrastructure such as vector databases; an RLM needs a sandbox that executes model-written code. The open-source rlm library supports local, Docker and several cloud sandboxes, and warns that its default local environment should not be used in production.

Trainability

RAG components are trained separately or jointly, and the practice is well understood. For RLMs the first results are recent: the paper's RLM-Qwen3-8B, post-trained on about 1,000 filtered trajectories, is reported to outperform its base model by 28.3% on average. This is a small-scale result from the method's authors.