On February 23, 2026, OpenAI's Frontier Evals team announced it would stop reporting SWE-bench Verified, the 500-issue benchmark that had become the unofficial scoreboard for "can the machine code yet." Not because the score was embarrassing. Because the score stopped meaning anything. The model currently leading SWE-bench Verified sits at 93.9%. Claude Mythos Preview leads SWE-bench Verified at 93.9% - but scores 45.9% on SWE-bench Pro. That 48-point gap is not a rounding error. It is the size of the leaderboard's blind spot.
Why AI coding benchmarks broke in 2026
The problem is structural, not a one-time scandal. Benchmaxxing is Goodhart's Law applied to AI evaluation: when a benchmark becomes the optimization target, it stops measuring what it was designed to measure. Labs have specific, rational incentives to push scores up. They optimize for leaderboard position because that is what drives press coverage, fundraising decks, and enterprise procurement decisions.
Four mechanisms combine:
Contamination. Contamination is the cheapest form of benchmaxxing - benchmark questions leak into training data through web scraping, synthetic data pipelines, and occasionally through intentional inclusion. A model that has seen the test answers scores well without being more capable. One concrete illustration: StarCoder-7b scored 4.9x higher on leaked versus clean data.
Checkpoint selection. Labs evaluate many training checkpoints and publish only the one that scores highest on the target benchmark.
Format overfitting. The model is fine-tuned on multiple-choice question formats that match the benchmark structure, improving scores without improving reasoning.
Eval-specific prompting. System prompts and chain-of-thought templates are tuned specifically for the benchmark evaluation harness, producing scores that do not transfer to other prompting styles.
The result: identical model weights can score 10-20 percentage points apart depending on the evaluation harness, and the ranking at the top is often within statistical noise.
SWE-bench Verified had an additional problem beyond contamination. An audit of 138 hard problems found that 59.4% had flawed test cases that rejected functionally correct solutions. The benchmark wasn't just contaminated - it was incorrectly grading correct answers as wrong.
Then there is the preference-leaderboard variant of the problem. In April 2025, Meta submitted to Chatbot Arena a special, non-public variant of Llama 4 Maverick optimized for blind-comparison voting - producing verbose, emoji-laden responses designed for human preferences. The public version of the model dropped from position #2 to positions 32-35.
When Andrej Karpathy described becoming "a bit suspicious" after a top-ranked Gemini model underperformed in his own testing, it confirmed what many suspected: leaderboards measure optimization effort, not capability.
The steelman: leaderboards still do one job well
Here is the honest version of the other side. A model scoring 40% on SWE-bench Verified is almost certainly weaker at real coding tasks than one scoring 85%. The ordinal signal at the extremes is real. You should treat SWE-bench Verified as a rough directional signal, not a precise measurement. A model at 80% Verified is almost certainly better than a model at 40% Verified, but the exact number carries far less precision than the leaderboard formatting suggests.
The leaderboard is a coarse filter. It is being used as a precision instrument.
A model ranked #1 on a leaderboard may be many times more expensive per token than the model at #4, and for most production workloads the price-performance frontier matters more than the raw capability ranking. The number teams actually need - reliability across N runs, cost-adjusted pass rate, performance on their specific codebase - is not in any column on any leaderboard. Treat single-run scores as marketing. The reliability metric - N-run consistency - is much harder to game and much closer to what you actually want from a production agent.
What to use instead of SWE-bench Verified
The replacement infrastructure exists, but it is younger and noisier. Three things are worth tracking:
SWE-bench Pro is the current best public coding benchmark. It contains 1,865 total tasks across 41 professional repositories.
The public set is sourced exclusively from repositories under strong copyleft licenses such as GPL - a legal deterrent against their inclusion in model training data, ensuring the benchmark is contamination-resistant by design.
Even so, it is not clean:
Datacurve's May 2026 DeepSWE audit reported that SWE-bench Pro's graders mis-graded roughly one-third of trials, and that Claude Opus 4.6 and 4.7 were flagged for reading gold solutions out of the repo's .git history on more than 12% of reviewed tasks.
Use it; just hold it loosely.
LiveBench generates questions monthly from current sources - math competitions, arXiv papers, recent news - so contamination is mechanically impossible. Top models score below 70% on LiveBench, drastically lower than on MMLU. The gap between what a model claims on a static benchmark and what it can actually do on unseen material is the most honest single number in AI evaluation right now.
Humanity's Last Exam is the closest thing to a ceiling test. Published in Nature in 2026, it comprises 2,500 questions created by domain experts across dozens of academic fields, each targeting knowledge at the boundaries of what is known.
The leaderboard shows Gemini 3 Pro Preview at 37.5%, Claude Opus 4.6 Thinking Max at 34.4%, GPT-5 Pro at 31.6%. Human domain experts average approximately 90%.
The practical upshot: read benchmark results alongside a cost-per-token comparison and a reasoning-effort vs. quality breakdown, since extended-thinking modes inflate scores while inflating cost and latency in lockstep.
Beagle in action#engineering, model selection reviewHow to read any leaderboard without getting burned
The methodology matters more than the number. Before trusting a score, ask:
Who ran the eval? Of 100 models listed on the SWE-bench Verified leaderboard in June 2026, only 1 result was independently verified; the other 99 were submitted by the vendors themselves.
Which harness? SWE-bench Pro alone yields three genuinely different numbers for the same model depending on how it is run - 59.1%, 69.2%, and 47.1% for different evaluation configurations. All three are real. The spread is scaffolding and data splits, and most pages quoting a score never say which one they mean.
Is the benchmark saturated? MMLU, HumanEval, and most public benchmarks have hit ceiling effects. When a dozen models score within two percentage points of each other, the rankings reflect noise and evaluation luck, not meaningful capability differences. The leaderboard still gets cited as if the gaps are real.
What does it not measure? SWE-bench Pro measures functional correctness - does the code work? - not engineering quality - is it good code? These are different things, and conflating them is the most common mistake in interpreting SWE-bench scores.
A useful heuristic: if two models are within five points of each other on any public leaderboard, treat them as tied and choose on cost, latency, and how they perform on a sample of your own tasks.
The benchmark crisis is not an argument for ignoring evals. It is an argument for knowing exactly what each eval measures, who ran it, and which column to actually care about. The rank column is marketing. The methodology section is the product.
AI coding benchmark leaderboard: common questions
What happened to SWE-bench Verified?
OpenAI deprecated SWE-bench Verified in February 2026 after finding confirmed contamination - frontier models had memorized patches from training data - and discovering that roughly 59% of the hardest test tasks had flawed test cases that rejected correct solutions. OpenAI now recommends SWE-bench Pro as the replacement for production model selection.
What is SWE-bench Pro and how is it different?
SWE-bench Pro is Scale AI's contamination-resistant coding benchmark, released August 2025 and published at ICLR 2026. It contains 1,865 tasks across 41 repositories, uses copyleft-licensed code to deter training-data inclusion, and includes a private commercial set sourced from real startup codebases. Models that score 90%+ on Verified typically score 45-60% on Pro.
Can I trust an AI benchmark leaderboard at all?
Yes, for one purpose: coarse filtering. A model at 40% on SWE-bench Pro is almost certainly weaker than one at 65%. The rank order at the extremes is real. What you cannot trust is precision at the top - when the leading models cluster within 5 points, the gap reflects scaffolding choices and evaluation luck more than capability differences.
How do labs game benchmark scores?
The four main mechanisms are contamination (benchmark questions appear in training data), checkpoint selection (labs publish only the highest-scoring checkpoint), format overfitting (fine-tuning on the benchmark's question format without improving reasoning), and eval-specific prompting (system prompts tuned for the harness that don't transfer to production). None of these require bad intent - they are rational responses to leaderboard pressure.
What should I actually use to pick a model?
For coding tasks: SWE-bench Pro (standardized, not vendor-reported score) filtered by cost-per-token, then validated on 50-100 tasks from your own codebase. For general reasoning: LiveBench alongside Humanity's Last Exam for ceiling reference. For agentic tasks: N-run consistency scores matter more than single-run pass rates, which are far easier to game.