On February 23, 2026, OpenAI quietly walked away from the most-cited number in AI coding. On that date, OpenAI's Frontier Evals team announced it would stop reporting SWE-bench Verified, the 500-issue benchmark that had become the unofficial scoreboard for "can the machine code yet."
The score had stopped meaning anything - and that single decision changes how every engineering lead should read a coding leaderboard.
The position I want to defend here: AI coding benchmark leaderboards are not broken in a fixable way. They are broken in a structural way. The scores vendors publish are a mix of genuine capability, training-data memory, and scaffold engineering - and those three things cannot be separated by reading a number on a table. That is not an argument for ignoring benchmarks. It is an argument for knowing exactly what you are reading before you act on it.
Why SWE-bench Verified stopped being an honest signal
SWE-bench Verified is a 500-task benchmark built from real GitHub issues pulled from open-source Python repositories. Since its release in August 2024, the industry used it widely to measure the progress of models on autonomous software engineering tasks, and it became a standard metric reported in frontier model releases. Early top scores sat around 20-30%. The benchmark felt like a genuine test.
Two things killed it: contamination and saturation, arriving roughly together.
OpenAI's Frontier Evals team stopped reporting SWE-bench Verified in early 2026 after an internal audit of 138 problematic tasks found that more than 60% were unsolvable as written due to flawed tests - and that frontier models could reproduce the gold-patch solutions verbatim from just the task ID, a clear fingerprint of training-data contamination.
Independent research backs the concern: one study found that 32.67% of successful SWE-bench Verified patches involved solution leakage, and that models recall the correct file paths from training data up to 76% of the time.
When a model can "solve" a third of the tasks partly by remembering the answer, a 90%-plus score is measuring memory as much as capability.
The saturation problem compounds this. Frontier models gained 30 percentage points in a single year on Humanity's Last Exam, a benchmark built to be hard for AI and favorable to human experts. Evaluations intended to be challenging for years are saturated in months, compressing the window in which benchmarks remain useful for tracking progress.
The contamination-resistant alternative, SWE-bench Pro, was built by Scale AI with privately authored tasks and a withheld commercial test split. The score gap between the two is the most telling number in this debate: a parallel leaderboard - SWE-bench Pro, designed explicitly to resist gaming - shows those same models scoring between 46% and 58%. Against SWE-bench Verified's 93%+, that is a 35-point contamination gap on the same underlying model weights.
The steelman: leaderboards are not useless, they are misread
Here is the honest version of the pro-benchmark case. Benchmarks are not useless. They give you a starting point and help filter out models that are genuinely far behind. A model that scores 30% on SWE-bench Pro probably cannot carry a real agentic coding task. A model at 55% might. That filtering job is real and worth having.
Benchmarks help measure progress. Leaderboards help compare systems. Reliability tells you whether those systems can actually be trusted. Those are three distinct jobs, and the mistake is using a leaderboard score to answer all three at once.
The 2026 AI Index found that the performance gap between the top US and Chinese models had narrowed to about 2.7% as of March 2026 on selected public measures. That is a genuinely useful geopolitical signal - at the right level of abstraction. It is not a verdict on which country leads in AI overall, which depends on research depth, talent, infrastructure, and supply chains that no leaderboard captures. Read narrowly, directionally informative. Read as a verdict, an overreach.
Procurement decisions, vendor pitches, and "which model should we standardise on" debates all lean on benchmark scores as if they were thermometer readings. They are closer to opinion polls: directionally useful, methodology-dependent, and trivially gameable by anyone motivated to do so.
The steelman lands here: leaderboards are a coarse filter that save you the work of testing fifty models. The mistake is letting the filter become the decision.
What the scaffold gap means for your evaluation
There is a second distortion that gets less attention than contamination: the harness multiplier. When two vendors report different scores for the same base model, the harness is usually the explanation. The scaffold - the agent loop, tool calls, retry logic, context management - can move a model's SWE-bench score by 20 points or more without changing the underlying weights at all. Claude Fable 5 tops SWE-bench Verified at 95%, but 99 of 100 results are self-reported and the scaffold gap can exceed 28 points.
This matters because you will run the model inside your own scaffold, not the vendor's. A score achieved with a bespoke eval harness tells you very little about what that model will do inside your CI pipeline or your Cursor workflow.
A model ranked #1 on a leaderboard may be many times more expensive per token than the model at #4, and for most production workloads the price-performance frontier matters more than the raw capability ranking. Read benchmark results alongside a cost-per-token comparison and a reasoning-effort vs. quality breakdown, since extended-thinking modes inflate scores while inflating cost and latency in lockstep.
What to actually do instead
The benchmark is not the eval. Your eval is your codebase.
| Signal | What it tells you | What it does not tell you |
|---|---|---|
| SWE-bench Verified | Model is not obviously weak; good shortlist filter | Whether the score reflects skill or memory |
| SWE-bench Pro | Cleaner capability signal across contamination-resistant tasks | How the model behaves on your stack |
| LiveCodeBench | Reasoning on post-cutoff problems, cutoff-anchored | Non-competitive-programming tasks |
| Your own task set | Ground truth for your workflow | How it generalises elsewhere |
Match the benchmark to your workflow. If your primary use case is bug triage and single-issue patches on a Python codebase, SWE-bench Pro is a reasonable signal. If you're doing multi-file feature work across a polyglot stack, FeatureBench and SWE-EVO scores are more relevant. If your engineers use agents inside a terminal-first workflow, Terminal-Bench performance matters.
The most honest thing a leaderboard can do is hand you a shortlist of five models worth testing yourself. After that, the eval is yours to run. Build 10-20 tasks drawn from real bugs or features your team has shipped, run each candidate model against them on your actual toolchain, and score by outcome - does the patch apply, do the tests pass, does a reviewer consider it correct? That is a small eval, and it will tell you more than any public leaderboard.
A teammate like Beagle - working from your Slack or Teams channel where engineering decisions get made - can surface the right benchmark sources and draft a model-comparison summary before a procurement call, so the conversation starts with data rather than vendor slides.
Beagle in action#engineering-tools, 10:02amAI coding benchmark leaderboards: common questions
What is the difference between SWE-bench Verified and SWE-bench Pro?
SWE-bench Verified is a public 500-task benchmark drawn from open-source GitHub issues; it is contamination-prone because the fixes predate most model training cutoffs. SWE-bench Pro uses privately authored tasks across 1,865 problems and a withheld commercial split, making it harder to game. The same frontier models score 46-58% on Pro versus 90%+ on Verified
- the gap is the contamination effect made visible.
Why did OpenAI stop reporting SWE-bench Verified scores?
OpenAI's manual audit of 138 o3 failures revealed 59.4% were caused by test flaws, not model limitations, leading them to recommend discontinuing SWE-bench Verified reporting. The deeper issue was evidence that frontier models could recall gold-patch solutions from training data rather than solve the tasks from scratch.
Can I trust Arena-style human preference rankings more than static benchmarks?
Arena leaderboards - where real users vote between anonymized model outputs - resist training-data contamination because tasks are live. They aggregate real user preference votes between anonymised models, but the failure modes are style bias, demographic skew, and overlapping confidence intervals at the top. They are best read with the confidence-interval column, not the rank. For coding tasks specifically, human preference is also a weak signal: users often prefer fluent-sounding wrong code over correct-but-terse patches.
Does a better benchmark score mean a model will cost more to run?
Not reliably. Grok 4.5 is the cheapest in the current top 10 at $2.00 per million tokens. Rankings and pricing are almost entirely uncorrelated, and the models leading on reasoning benchmarks often use extended-thinking modes that inflate both score and per-call latency. Always run the cost math against your actual call volume.
What benchmark should a team use if they just want a quick model shortlist?
Use SWE-bench Pro for coding agents, LiveCodeBench for reasoning tasks dated after your model's training cutoff, and run at least a handful of your own real tasks before committing. No single model wins every benchmark
- the winner depends entirely on which dimension you are optimising for.