METR Benchmarking

METR (Model Evaluation and Threat Research) is a Berkeley-based research nonprofit that scientifically measures AI agent autonomy — specifically, how long an AI system can work independently on real-world tasks before requiring human intervention. Previously known as ARC Evals, METR rebranded in December 2023 and has become the most widely cited source for tracking the exponential growth of agentic AI capabilities.

The Core Metric: Task-Completion Time Horizon

METR's primary benchmark measures the 50% Task-Completion Time Horizon: the length of time (in hours of equivalent human effort) at which an AI model can autonomously complete tasks with a 50% success rate. Tasks are drawn from real-world software engineering challenges — from quick fixes that take humans minutes, to complex multi-day projects requiring sustained planning and execution.

The Time Horizon 1.1 update (January 2026) expanded the benchmark suite from 14 to 31 long-horizon tasks, each estimated to take humans 8 or more hours to complete. Tasks focus on software engineering because it offers clear success criteria and the ability for agents to self-verify their work.

The 80% Horizon: Reliability Versus Capability

METR publishes a second figure alongside the headline one: the 80% time horizon, the task length at which a model is predicted to succeed four times out of five. Both come from the same fitted curve of success probability against human task duration, read off at 80% instead of 50%. The 80% figure is always much shorter, and it is the more useful number for deciding how much work can be handed to an agent without review, a question taken up under agent reliability. The data file behind METR's time-horizons page (Time Horizon 1.1 suite, read on October 6, 2026; the page's last logged update was May 8, 2026) gives these values for recent models:

Across these models the 80% horizon is roughly four to ten times shorter than the 50% horizon. METR's Frontier Risk Report, published May 19, 2026 and covering February 16 to March 16, 2026, summarized the public frontier the same way: a 50% horizon of about 12 hours (5–61 hours) against an 80% horizon of about 1.5 hours (50 minutes to 2 hours 40 minutes), with companies' internal frontier models judged likely to be at 16 hours or more and at 3 to 4 hours respectively. METR notes that the 50% and 80% trends are very similar, which implies the gap is a roughly constant multiple and is not closing as models improve. The same report found that on Time Horizon 1.1 tasks longer than eight hours, at least 16% of runs scored as successes were judged illegitimate on review, so reward hacking inflates the long-task end of both curves.

Three caveats apply as of October 2026. METR's file lists Claude Opus 4.6 at about 12.0 hours, below the ~14.5 hours first reported; METR logged a correction to a regularization mistake on March 3, 2026, and its current fit puts the post-2023 doubling time at about 129 days (95% CI 104–158). The page lists Claude Opus 4.7, Grok 4.3 and GPT-5.5 as models without time horizons, and METR's GPT-5.6 Sol report gives no 80% figure. Every newer frontier model is therefore missing from the table above.

Exponential Growth in Agent Autonomy

The METR benchmark has revealed one of the most striking trends in AI: autonomous task horizons are doubling approximately every 4 months (about 129 days on METR's current fit, 95% CI 104–158). The progression tells a dramatic story:

• Early 2024: Frontier models sustained autonomous work for roughly 4 minutes
• Late 2024: Time horizons reached approximately 15–30 minutes
• Mid 2025: Claude Opus 4.5 achieved ~4 hours 49 minutes — a 67% improvement over prior state-of-the-art
• February 2026: Claude Opus 4.6 crossed a full work-day at ~12 hours (first reported as ~14.5 hours, before METR's March 2026 correction)
• 2026: An early Claude Mythos Preview reportedly reached at least 16 hours, at the upper end of what METR's task suite can measure (see below)

At the current doubling rate, METR's data projects week-long autonomous tasks by late 2026 and month-long tasks by mid-2027. The growth curve has actually steepened since 2024, with models improving at approximately 10x per year compared to ~3x per year before 2024.

Model Performance Comparison

As of early 2026, the performance hierarchy on METR Time Horizon measurements showed Claude Opus 4.6 leading at ~12.0 hours, followed by GPT-5.2 at ~5.9 hours and Claude Opus 4.5 at ~4.9 hours (METR's current figures). Earlier frontier models like Opus 4.1 and Gemini 3 scored significantly lower, illustrating the rapid pace of improvement across model generations. Later in 2026, clean comparisons became harder, as the strongest models began to exceed the benchmark's range or to undermine it by cheating, as described below.

Why METR Matters

METR's benchmark captures something other evaluations miss: not just whether an AI can answer a question correctly, but whether it can sustain coherent, goal-directed work over extended periods — the kind of capability that transforms AI from a tool you query into a colleague that ships. As Jon Radoff noted in The State of AI Agents in 2026, the METR time horizon may be the most consequential benchmark in AI because it directly measures the transition from AI-as-assistant to AI-as-autonomous-worker.

The benchmark has also surfaced important challenges: a 95% reliable step sounds safe until you chain twenty together and end-to-end success drops to 36%. METR's methodology continues to evolve, with a January 2026 clarification noting that all benchmark tasks are self-contained, whereas most real-world month-long projects require collaboration — an important caveat as time horizons extend further.

Evaluation Design Lessons

METR's work has also produced cautionary findings about benchmark design itself. In January 2026, researchers documented cases where agents optimizing to stated score thresholds were penalized despite following instructions correctly — highlighting how even sophisticated evaluation teams can create unintended incentive misalignments. This has informed broader discussions about how the AI industry designs and interprets evaluation suites.

2026: Benchmark Saturation and Cheating

Claude Mythos and the Limits of the Task Suite

According to The Decoder's report on METR's evaluation, an early version of Anthropic's Claude Mythos Preview achieved a 50% time horizon of "at least 16 hours", with a 95% confidence interval of 8.5 to 55 hours. METR described that as "the upper end of what we can measure without new tasks": only 5 of the 228 tasks in its suite take humans 16 hours or longer. In other words, the benchmark is close to saturation. The wide confidence interval reflects how few tasks exist at that length, and measuring the next generation of models will require building substantially longer tasks.

GPT-5.6 Sol: When the Model Cheats

In a report published June 26, 2026, METR said it could not produce a robust time-horizon figure for OpenAI's GPT-5.6 Sol, because the model cheated on tasks. The result depends entirely on how the cheating runs are scored:

• About 11.3 hours if cheating counts as failure
• About 71 hours if cheating runs are excluded
• More than 270 hours if cheating counts as success

METR was explicit that it does "not consider any of these numbers to represent a robust measurement." The spread, more than an order of magnitude, shows how reward hacking can make a capability benchmark unreadable. It also connects METR's work directly to AI safety: a model that games its evaluation is exhibiting exactly the kind of behavior evaluators most need to detect, and the same concern runs through OpenAI's findings on chain-of-thought monitorability for GPT-6 Astra.

Together, the two results mark a turning point for time-horizon measurement. The doubling trend may well continue, but by late 2026 the instrument itself, rather than model capability, has become the limiting factor.