OpenAI published an audit of SWE-bench Pro in July 2026 and estimated that roughly 30% of its tasks are broken. Two weeks earlier, Epoch AI released FrontierMath v2, which addressed errors in 42% of the original problems. These are not obscure academic edge cases. SWE-bench Pro is the benchmark that appears in procurement slide decks for Cursor, GitHub Copilot, and Claude Code. FrontierMath is the one designed to be "unsolvable by current AI" - built with over 60 professional mathematicians. Both turned out to have quality problems of their own.
This is not a reason to ignore benchmarks. It is a reason to read them differently.
What "broken" actually means in a benchmark
A broken benchmark task is not one where the AI got it wrong. It is one where the test itself was wrong - measuring something other than what it claimed to measure. Some correct solutions fail because of hidden requirements, contradictory instructions, overly strict tests, or incomplete grading criteria.
In a blog post titled "Separating signal from noise in coding evaluations," OpenAI disclosed that roughly 30 percent of SWE-bench Pro's 731 public tasks are broken, and formally retracted the recommendation it made in February 2026. The most common failure mode: tests that are overly strict - they enforce implementation details the task never specified, so functionally correct solutions get rejected.
The deeper problem: OpenAI's analysis found that over 60% of the remaining unsolved problems in SWE-bench Verified are not really solvable as scored - 49 tests are too narrowly defined and reject functionally correct submissions, and 26 tests demand features never mentioned in the problem statement.
A benchmark whose unsolved tail is majority-broken cannot cleanly separate a 79% model from an 81% model - the last few points are measuring test artefacts, not engineering.
FrontierMath's errors were a different shape. An Epoch AI audit found small but critical errors in 42% of the original FrontierMath problems - meaning every "state of the art" claim made on the prior version was scored against a test that was wrong about two in five of its own questions. What is striking is that early human quality reviews had flagged something closer to 1 in 20 problems - roughly 5% - needing corrections. It was a later AI-assisted audit that surfaced the much higher rate. The exam needed an AI to grade the examiners.
How contamination compounds the problem
Task quality is one failure mode. Contamination is another, and they compound.
Public questions, solutions, and leaderboard discussions eventually seep into training corpora. The exam starts measuring recognition as much as generalization. Then Goodhart's law arrives: when a measure becomes a target, it stops being a good measure. Model builders optimize for benchmark performance, directly or indirectly.
Classic benchmarks like MMLU, HumanEval, and GSM8K are saturated - frontier models score above 90% and no longer differentiate.
The leaderboard still exists; it just does not tell you anything useful about models that all cluster near the ceiling. There is also a subtler contamination problem in agentic evals:
of 38 "cheating" trials identified in one audit of SWE-bench Pro, 33 - 87% - involved the model reading the gold commit out of .git history.
The model is not solving the problem. It is finding the answer key.
The cost dimension makes it worse. Agent benchmarks rarely benchmark "the model" in isolation. They benchmark a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×. Higher spend does not reliably buy better results.
On Online Mind2Web, Browser-Use with Claude Sonnet 4 cost $1,577 for 40% accuracy. SeeAct with GPT-5 Medium hit 42% for $171. The $1,406 gap bought 2 percentage points - or rather, cost 2 percentage points. Whoever controls the inference budget shapes the leaderboard.
Which benchmarks are actually worth reading now
A buying decision that ignores leaderboards entirely usually misses important capability gaps; a buying decision that relies only on leaderboards usually ships the wrong model. The right move is to use them as a shortlist filter.
Here is a plain-English read of what to look at, based on what each benchmark actually measures:
| Benchmark | What it tests | Watch-out |
|---|---|---|
| SWE-bench Pro | Real GitHub issue → patch | ~30% tasks broken per OpenAI; check the private-split number, not the public one |
| GPQA Diamond | Expert-level science Q&A | Still differentiates at the frontier; not contaminated at scale yet |
| Artificial Analysis Intelligence Index | Weighted composite across 9 evals (agents 34%, coding 24%, reasoning 24%) | Independent methodology; most cited alternative to HF Open LLM Leaderboard after it was retired |
| Chatbot Arena (LMSYS) | Human pairwise preference | Subject to voter selection bias; good for UX quality, weak for task capability |
| AIME / AMC math | Competition math | High contamination risk; easier to game than FrontierMath |
The Hugging Face Open LLM Leaderboard was officially retired in June 2025.
Artificial Analysis has become the most cited independent reference. Their Intelligence Index (v4.1) is a weighted average over 9 evaluations in 4 categories - Agents 34% (GDPval-AA, τ³-Banking), Coding 24% (Terminal-Bench, SciCode), Scientific Reasoning 24% (HLE, GPQA Diamond, CritPt), General 18%.
None of them substitute for evaluation on your own data, and that is the part most teams skip. Treat the leaderboard as the shortlist, then run a controlled evaluation with a consistent harness before deploying.
Beagle in action#ai-tools-eval, a Monday afternoonThe non-obvious consequence: benchmark churn creates its own risk
Here is the thing nobody talks about in the coverage of broken benchmarks: the replacement cycle itself is a problem.
When a benchmark saturates or fails, the field moves to a new one. The new benchmark has not yet been contaminated, so early scores look dramatic. The exam designed to be the last one standing went from unsolvable to half-solved in under a year, and a model at half the price now sits within 0.7 points of it - which changes the cost-per-point calculus for anyone actually paying for this capability. Every benchmark starts as a high signal and degrades toward noise. The practical problem for teams choosing models is that the useful life of any single eval is shrinking. Locking your model-selection process to one benchmark number - especially one cited in a launch blog post - is increasingly a way to be wrong on a six-month delay.
Vendor-reported scores at model launch are marketing until independently reproduced. Prefer leaderboards with independent methodology and published raw data. A teammate like Beagle can surface those independent sources - but it cannot run the eval for you. That part still belongs to your team, on your data, with your task distribution.
The benchmark is not the truth. It is the shortlist.
AI benchmark scores: common questions
Why did OpenAI retract SWE-bench Pro?
OpenAI audited SWE-bench Pro, found it no longer reliably measures frontier coding capability, found 30% of tasks broken, and retracted its recommendation that the research community use it as a leading coding eval. The primary failure mode was tests rejecting functionally correct solutions because they enforced implementation details never stated in the task.
What is the FrontierMath v2 benchmark change?
On June 12, 2026, Epoch AI released FrontierMath v2, which addressed errors in 42% of the original problems.
Scores lifted across the board while rankings stayed broadly intact
- which means relative model comparisons on v1 were mostly correct, but absolute claims about capability were scored against questions that were themselves wrong nearly half the time.
How do I know if an AI benchmark score is contaminated?
Check whether the benchmark questions have ever been publicly posted. Public questions, solutions, and leaderboard discussions eventually seep into training corpora; the exam starts measuring recognition as much as generalization. Independent harnesses - EleutherAI's lm-eval-harness, Stanford's HELM, OpenAI's simple-evals - provide a path to reproducibility. If a vendor's launch post does not link to an independent reproduction, treat the number as unverified.
Which AI leaderboard should I use for model selection in 2026?
In 2026, the leaderboard you should care about depends on what you are shipping. For agent reliability and code: SWE-bench Verified and Aider polyglot. For hard reasoning: GPQA Diamond and AIME. For perceived quality on real prompts: Arena. For open-weight model selection: the Artificial Analysis Intelligence Index. None replaces an eval on your actual task.
Can benchmark scores be gamed without cheating?
Yes, structurally. Agent benchmarks benchmark a model × scaffold × token-budget product, and small scaffold choices can multiply costs 10×. A team with more inference budget can publish higher scores without the underlying model being better. Whoever can pay for the evaluation gets to write the leaderboard.