On February 23, 2026, OpenAI published a short post announcing it would stop reporting scores on SWE-bench Verified - the 500-task coding benchmark that had been the industry's unofficial answer to "can this model actually write software?" The reason was not embarrassment. Improvements on SWE-bench Verified no longer reflected meaningful improvements in real-world software development ability. Instead, they increasingly reflected how much the model was exposed to the benchmark at training time.
That single sentence is worth reading twice before you look at any leaderboard again.
What broke, exactly
SWE-bench Verified launched in August 2024 as a human-curated subset of 500 real GitHub issues. A curated set of 500 real GitHub issues, human-verified for correctness, it promised a reliable signal of how well models can actually fix bugs and implement features. For about eighteen months it was the number every model release cited. Then OpenAI did something unusual: it audited the benchmark that it had helped create.
OpenAI audited 138 problems - 27.6% of the 500-problem dataset - that its o3 model couldn't consistently solve across 64 independent runs. The findings were damning: 59.4% of these problems had material issues in test design or problem descriptions. Some tests required a specific function name never mentioned in the issue description. Others checked for behavior pulled from the pull request, not from anything in the bug report.
The contamination side was worse. OpenAI built an automated red-teaming system using GPT-5 to probe competing models for benchmark knowledge. The results showed all tested frontier models could reproduce original human-written solutions or quote verbatim problem details they should never have seen. Give a model a task ID and a small hint, and it quotes the gold patch back to you - because the patch was in its training data all along.
The consequence: models scoring 80% on Verified dropped to roughly 23% on SWE-bench Pro - a benchmark designed to resist contamination. That 57-point gap does not represent a capability difference. It represents the distance between knowing the answer and having worked it out.
A benchmark whose test set lives on GitHub gets crawled into the next pretraining run. After that, a high score has two equally plausible explanations: the model reasoned its way to the fix, or it retrieved the fix from memory. The number alone cannot tell you which.
Why every major leaderboard has a version of this problem
SWE-bench is not special. It is just the example where the keeper of the benchmark said so out loud.
MMLU and MMLU-Pro are functionally saturated above 88% for frontier models, making score differences at the top statistically meaningless.
A review found invalid question rates ranging from 2% on MMLU Math to 42% on GSM8K. Those are not rounding errors. A benchmark where nearly half the questions may be invalid cannot cleanly separate two models.
Research has shown model rankings are sensitive to prompt format perturbations
- meaning a model can move several positions on a leaderboard based on how its input is formatted, without any underlying capability change. Agent scaffolding alone inflates SWE-bench scores by 12+ points. The same model, different harness, different score.
Arena (formerly LMArena, formerly Chatbot Arena) takes a different approach: it ranks models through pairwise blind voting. A user enters one prompt, two anonymous models answer it, and the user votes for the better response. Arena then feeds those results into a Bradley-Terry rating system, similar to Elo for pairwise competitions. With nearly 6 million votes cast across hundreds of models, the signal is harder to fake than a static test set. But it carries its own blind spots. Separate research suggests that Arena leaderboard standing may partly reflect adaptation to the platform rather than general capability.
A model can be confidently wrong and still win votes for sounding good. Arena measures preference, not truth.
The benchmarks still worth watching
The answer is not to throw out all evals. It is to know what each one defends against.
LiveCodeBench is the clearest illustration of the idea. It pulls problems from competitive programming sites, each stamped with a publication date, and lets you score a model only on problems released after its training cutoff. The slider is the whole defense: drag it past the cutoff and you are testing reasoning on problems the model could not have memorized.
GPQA Diamond asks graduate-level science questions curated by domain experts across physics, chemistry, and biology. In 2026, GPQA Diamond, Humanity's Last Exam, SWE-Bench Verified, and LiveCodeBench are the four tests that actually separate frontier models because they resist data contamination and reward genuine reasoning over pattern recall
- though SWE-bench Verified has now earned its asterisk.
Humanity's Last Exam is the current high-water mark for difficulty. It is the current ceiling for closed-ended AI evaluation, comprising 2,500 questions created by domain experts across dozens of academic fields, each targeting knowledge at the boundaries of what is known.
Human domain experts average approximately 90% on it. The scores are low enough, and the test set private enough, that contamination has not yet eaten the signal.
For coding specifically, OpenAI now points to SWE-bench Pro. Pro's 1,865 tasks average 107 lines across 4.1 files, minimum 10-line changes, with over 100 tasks demanding 100+ line modifications. It tests multi-file reasoning on enterprise-grade complexity.
Scale built it with contamination resistance: strong copyleft licenses discourage commercial training inclusion, and tasks come from 11 public repositories plus commercial codebases from real startups.
How to actually use a leaderboard
Treat the number as a shortlist filter, not a decision. When five models from four labs sit statistically tied near 80% on a benchmark, the benchmark has stopped discriminating at the frontier. A 3-point gap between models you are evaluating is almost certainly within noise.
The companies publishing these benchmarks are also the ones selecting which benchmarks to publish. So the first thing to check is who ran the eval - the lab selling the model, an independent team using the same harness as everyone else, or a live platform updated with problems the model could not have trained on. Those three categories carry very different epistemic weight.
Enterprise agentic AI systems show a 37% gap between lab benchmark scores and real-world deployment performance. That number is large enough to matter when you are committing budget. The only evaluation that closes the gap is one run on your own tasks, your own data, your own definition of a correct answer.
A teammate like Beagle sees this play out at the workflow level: teams that pick a model from a leaderboard screenshot and move straight to production tend to swap models within sixty days. Teams that spend a week running the model against their actual Slack threads, documents, and edge cases tend not to.
Real-world performance in your environment matters more than leaderboard position. The 59% failure rate OpenAI found should make us skeptical of all automated evaluation - human judgment and real-world testing remain the most reliable signals.
The SWE-bench decision was not a crisis. It was a lab admitting out loud what anyone reading the fine print already suspected. The leaderboard number you used to justify a procurement call was partly measuring memory, not skill. Knowing that is not a reason to stop reading evals. It is a reason to read them with the same skepticism you would bring to any other marketing material - which is exactly what they are, until proven otherwise.