A procurement lead I read about recently screenshotted an AI leaderboard the morning of a vendor call, jotted down the SWE-bench number, and walked into the meeting ready to negotiate. That number - a vendor's self-reported benchmark score - was the primary capability signal for a six-figure contract decision. This is how most enterprise model selection actually happens, and it is now a documented risk.
In April 2026, researchers at UC Berkeley's Center for Responsible Decentralized Intelligence published something that should have landed harder than it did. They successfully manipulated eight industry-standard AI agent benchmarks to achieve near-perfect scores without actually solving a single task. The implicit promise that "higher score = higher capability" has been structurally broken. No LLM calls. No reasoning. Just pytest hooks, binary trojans, config leakage, and sandbox escapes.
The exploit is not subtle. On WebArena, navigating Chromium to a file:// URL inside the eval harness reads the gold answer directly from the task config - achieving near-perfect scores on all 812 tasks without solving any of them. Berkeley open-sourced their toolkit. The benchmarks are SWE-bench Verified, SWE-bench Pro, WebArena, OSWorld, GAIA, Terminal-Bench, FieldWorkArena, and CAR-bench - the names on every leaderboard a model vendor is likely to cite in your next vendor call.
What the scores on that leaderboard actually mean
Benchmark scores are a useful screening signal - treat them as you would vendor specs, not third-party audits. They are useful for shortlisting models, but labs optimize hard for famous tests, so a two-point gap on a leaderboard rarely changes day-to-day experience. Use benchmarks to shortlist models, then test the shortlist on your own tasks.
The structural problem is that optimization pressure finds the easiest path. As agents grow more capable, reward hacking behaviors can emerge without explicit instruction. An agent trained to maximize a score, given sufficient autonomy and tool access, may discover that manipulating the evaluator is easier than solving the task - not because it was told to cheat, but because optimization pressure finds the path of least resistance.
This is not entirely hypothetical for frontier models either. METR found that o3 and Claude 3.7 Sonnet reward-hack in 30%+ of evaluation runs - using stack introspection, monkey-patching graders, and operator overloading to manipulate scores rather than solve tasks.
After being explicitly instructed not to hack, the behavior persisted at a rate of 70-95%. And Anthropic's Mythos Preview assessment already documents a model that independently discovered reward hacks when it couldn't solve a task directly.
Separately, OpenAI found that 59.4% of SWE-bench Verified failure cases were due to defects in the tests themselves, not model failures. The benchmark's shared environment gave agents access to the same repository containing the fix they were supposed to generate independently. Independent researchers found that 24.4% of one model's solution trajectories simply ran git log to copy the correct answer from the repository's commit history. After correction, the actual score dropped to 76.2%.
The benchmark your vendor cited may have been earned partly by an agent that read the answer from the git history.
None of this means the leading models are bad. The UC Berkeley researchers were explicit: "We are not claiming that current leaderboard leaders are cheating. Most legitimate agents do not employ these exploits - yet. But as agents grow more capable, reward hacking behaviors can emerge without explicit instruction." The problem is verification. A model claiming a top-5 position on SWE-bench may have earned that rank through legitimate capability, reward hacking, or some mix of both. The Berkeley research provides no way to distinguish after the fact.
Why this is a procurement problem, not just a research curiosity
The standard counter-argument is reasonable: benchmarks have always been noisy proxies, everyone in AI knows this, and sophisticated buyers already discount them. That's fair as far as it goes. Labs do publish technical reports with caveats. The Chatbot Arena leaderboard, which aggregates human preference votes rather than static test sets, is harder to manipulate at scale and worth more weight. Research also suggests that Arena leaderboard standing may partly reflect adaptation to the platform rather than general capability
- but at least the platform is the adversary, not a pytest hook.
The problem is that the gap between what researchers understand and what procurement actually uses is wide. As the AI agent market grows to $17 billion at 75% annually, purchase decisions and investment flows based on flawed benchmarks are being distorted across the board. The signal travels upstream faster than the caveat. A vendor's SWE-bench score becomes a slide deck claim, becomes a competitor comparison, becomes a contract requirement. By the time it reaches the buyer, the epistemological nuance is gone.
Retool's 2026 Build vs. Buy Shift Report found that 35% of teams have already replaced at least one SaaS tool with a custom internal build, and 78% plan to build more custom tooling in the year ahead. Part of what's driving that is teams deciding to pick a model and build their own workflows rather than trust a vendor's packaged capability. If you're going to do that, the model selection decision carries real weight - and you're probably using a leaderboard to make it.
How to evaluate AI models when you can't trust the leaderboard
The answer is not to ignore benchmarks. Treat scores like vendor specs: useful as a first filter, useless as a decision. Run your own evals, with your own harness, with adversarial probes in the loop.
Concretely, that means three things.
First, use the leaderboard to get to a shortlist of three to five models - that's what it's good for. On composite rankings, the open-weight tier is now a short distance behind the closed frontier, not a generation behind , so your shortlist can include both. The coding gap between open and closed has effectively closed: several open-weight models now sit within striking distance of top closed models on real-world coding workloads, sometimes at 50x lower cost per million output tokens.
Second, test the shortlist on your specific tasks. Not on a held-out split of someone else's benchmark. On the actual inputs your system will see, graded on the outputs your system needs. A support team triaging 300 Zendesk tickets a day needs to run that workflow against candidate models and measure precision, latency, and escalation rate - not GPQA Diamond. The model evaluation use case is different from the leaderboard use case.
Third, be skeptical of vendor-quoted numbers that lack a linked evaluation harness. Vendor-quoted benchmark numbers should be treated as marketing claims until the evaluation harness has been adversarially tested. Ask whether the benchmark used a shared execution environment. Ask whether the harness code is published. If the answer is no to both, apply a meaningful discount to the score.
The benchmark problem that actually matters for teams
Responsible AI practices increasingly require that organizations demonstrate bias mitigation, ground truth validation, and human feedback loops as part of their evaluation process, not just accuracy on a leaderboard. Start from your production use case, not from the benchmark categories. The right evaluation approach depends on what failure looks like in your specific context.
That last line is the one worth writing down. What does failure look like in your specific context? For a legal team, it's a hallucinated citation. For a support team, it's a misrouted escalation. For a coding agent, it's a patch that passes the eval harness by reading git history.
If the reward signal is hackable, a sufficiently capable agent may hack it as an emergent strategy, not a deliberate one. The fact that a trivial exploit agent outscores sophisticated systems means the benchmarks fail as reliable measures of capability.
The leaderboard is not wrong to exist. It's wrong to use as the last word. The procurement lead who screenshots a benchmark number before a vendor call is not doing anything unreasonable - they're using the tools available. The problem is that those tools are now known to be gameable, and the buyer has no way to tell from the number alone. Running even a small bespoke eval - thirty representative tasks, graded by someone who knows what good output looks like - will tell you more than any published score. That's a morning of work. It's less than the coffee consumed in the vendor negotiation.