METR: the independent yardstick for AI agents

METR — Model Evaluation & Threat Research, pronounced "meter" — is a Berkeley nonprofit that tests what frontier AI systems can actually do. It began in 2022 as ARC Evals, the evaluation team of the Alignment Research Center, and went independent in December 2023. It evaluates models for Anthropic and OpenAI before release, unpaid, and takes no money from AI companies or their staff — though the labs do supply free tokens. METR says all of this itself, including that Anthropic reviewed its summary of Claude Opus 5.5 before publication. That habit of disclosure is the reason to read it.

The chart everyone quotes. METR's best-known metric is the time horizon: the length of task, measured in skilled-human time, that an agent completes with 50% success. On its suite of 228 software, machine-learning and security tasks it has grown exponentially for six years — doubling roughly every seven months since 2019, and every 4.3 months since 2023. By early 2026 the best public models sat around 12 hours.

Read it the way METR does. By METR's own account, a time horizon is not how long an AI can work unsupervised. A 50% horizon of X hours doesn't mean you can delegate tasks under X hours — at 80% success, the same models managed about 1.5 hours. Error bars run roughly a factor of two each way, the tasks are cleaner than real work, and visual computer-use tasks score 40–100 times lower. The best models now saturate the suite: anything above 16 hours is unreliable, and on tasks over eight hours at least 16% of "successes" turned out to be cheating.

The study that measured instead of asking. In 2025 METR ran a randomized trial: 16 experienced open-source developers, 246 real issues. With AI allowed they took 19% longer — and afterwards still believed AI had made them 20% faster. A late-2025 follow-up with 57 developers suggests they're faster now, but METR calls its data "only very weak evidence" of by how much: so many developers refused to work without AI that the control group itself became the problem. Its 2026 survey of 349 technical workers reports a median 1.4–2x gain in the value of their work — with METR's own warning that survey results "are not necessarily grounded in reality".

Our read: the trend is real, and it's the right reason to start now. But borrow METR's discipline before its headline. Set the success rate a task actually needs, not 50%. Measure the workflow before and after instead of asking how it feels — in 2025 the gap between perception and stopwatch was about 40 percentage points. It's the same baseline argument we made in our reading of Anthropic's agent report, and it's where an AI readiness assessment starts.