Scores on SWE-bench Verified - the go-to benchmark for measuring AI coding ability - plunged from roughly 70% to 23% when OpenAI moved to a contamination-resistant replacement. The old number, it turned out, no longer told you anything useful about whether a model could actually write software. That's not a footnote. It's the central problem with AI evals: the score and the capability it's supposed to represent can quietly come apart, and you won't notice until production bites you.
If you've ever picked a model off a leaderboard and found it disappointing on your actual workload, this is probably what happened. Understanding how evals work - and where they break - is the difference between a well-grounded model choice and one that just looks good on paper.
What an eval actually is, and how it differs from a benchmark
The terms get used interchangeably, but they describe different things. Benchmarks are a standardized testing approach with fixed question sets - every model receives the same test, answers are automatically graded, and results are reported as a single percentage. AI evals are a broader category: they assess how models behave in complex, real-world scenarios.
Similar to how traditional software teams write unit tests to ensure components behave as expected, AI teams write evals to test whether models produce outputs appropriate for the scenario. The analogy is useful but imperfect. A unit test checks a deterministic function. An eval checks a probabilistic system whose outputs can vary across identical inputs, and where "correct" is often subjective.
Evals are systematic, task-driven measurement exercises designed to test how well LLM-based systems perform on specific, operationally relevant tasks. Unlike public benchmarks, evals are typically tailored to a company's unique needs - transforming vague questions ("Is this model good?") into measurable, reproducible answers ("Under these specific conditions, the model achieves X performance with Y confidence").
A concrete example: a support team routing 400 Zendesk tickets a day might run an eval where they feed the model 200 labelled historical tickets and score how often it routes to the right queue. That's a task eval. MMLU, by contrast, is a public benchmark - 57 academic subjects, multiple-choice, same questions for every model, graded automatically. Both are evals in the loose sense. Only one tells you anything about the support team's actual use case.
Evals fall into three broad categories: capability evals that test what the model can do (task completion, tool use, planning); alignment evals that test whether it behaves as intended without being sycophantic, deceptive, or overly cautious; and safety evals that probe for harm.
The contamination problem that inflated every leaderboard
The most corrosive issue in public benchmarks is data contamination: evaluation datasets overlap with pre-training corpora, which inflates performance metrics and undermines the reliability of model evaluations.
On MMLU, roughly 29.1% of items showed contamination signals; on the Chinese benchmark C-Eval, 45.8% did. Meta's own Llama 2 report found that 16% of MMLU items overlapped with its training data - some severely, with more than 80% of tokens matching.
In its GPT-4 technical report, OpenAI disclosed that across 34 academic and professional exams it checked directly, 9 had more than 20% of items overlapping with training data.
Contamination happens when benchmark questions, or text closely derived from them, end up in a model's training corpus - so the model recalls the answer rather than reasoning to it. The result is that leaderboard rankings reflect, at least partly, which model has seen the most test questions during training. OpenAI's Frontier Evals team reported that models can reproduce the original gold patches or problem statements verbatim from the evaluation, using nothing but the task ID - an effect that touches all tested frontier models. When the answer is memorisable from an identifier, the score stops measuring software-engineering ability and starts measuring recall.
The industry response has been to keep building new, harder benchmarks - and then watch those get contaminated in turn. Saturation happens when frontier models cluster so tightly near the ceiling that the benchmark loses its ability to tell them apart. Both contamination and saturation turn a once-useful eval into a number that no longer measures what its name implies.
LLM-as-judge: the clever workaround and its own failure modes
Automated grading works fine when there's a clear right answer - a multiple-choice question, a regex match, a passing test suite. It breaks down when you're evaluating open-ended outputs: summaries, reasoning traces, tone, factual accuracy in a long document.
The industry landed on a pragmatic fix: use another LLM to grade outputs. The community has increasingly adopted LLM-as-a-judge approaches, which use LLMs themselves to assess model outputs - methods that tend to be more effective at capturing content-related nuances and generally achieve higher correlations with human judgments.
Tools like OpenAI Evals and Inspect exemplify lightweight evaluation harnesses with reusable templates - including model-graded judging - enabling rapid construction of task-specific tests while keeping data schemas and scoring protocols consistent across runs.
But LLM judges carry their own biases that are now well-documented. Among the various types of bias, some are consistent across all LLM judges: a tendency to prefer longer, authoritative-looking, and well-formatted responses. A model can score highly simply by being verbose and structured, even if its actual answer is weaker. GPT-4 gave higher scores to AI summaries than to human ones - even when humans actually preferred the human-written versions. This suggests that LLMs might use the same internal standards for both writing and judging, creating a self-reinforcement risk.
There's also egocentric bias: models tend to score their own outputs higher. The reliability of LLMs as evaluators may vary depending on the task, the LLM judge used, and the evaluation strategy employed.
Combining LLM evaluations using LLM-as-a-judge with human review is the current recommendation for scalable and trusted evaluation pipelines. Human review remains the ground truth - it's just too expensive to do at scale, so the practitioner's art is figuring out which fraction of outputs to route to humans versus automated scorers.
How to run evals that actually tell you something
The deeper problem is that most teams treat evals as a one-time gate rather than a continuous instrument. There are two key types of AI evals used at different points in the development process: pre-deployment validation (done during development to measure the impact of design and architecture choices) and production evaluations (continuous monitoring after deployment to detect issues with outputs).
The instinct to trust a pre-deployment benchmark score and call it done is understandable - and usually wrong. Don't assume strong benchmark performance means your application works well. A model might excel at general reasoning but fail on your domain-specific tasks.
Even though standardized benchmarks are useful, it is important to create specific test scenarios that are as close as possible to the real application of the LLM - enabling evaluation in the conditions actually expected in practice.
What that looks like concretely: take 100-200 real inputs from your production logs (anonymised), label the outputs you'd actually want, and run both the current model and any candidate replacement against that set. Score automatically where you can - exact match, structured output correctness, refusal rate on edge cases - and route the ambiguous 20% to a human reviewer. Then track that set across model updates, not just at release.
A mature eval practice - with versioned datasets, rigorous annotation, statistical rigor, and clear reproducibility controls - is essential for shipping safe, reliable AI products. The "versioned datasets" part matters more than it sounds. If you keep adding new cases to your golden set without versioning, you lose the ability to compare scores across time, because you're no longer running the same test.
The useful mental shift is to stop asking "what score did this model get?" and start asking "on which inputs did it fail, and do those inputs look like mine?" A study by MDPI found that prompt selection alone can cause performance differences of up to 76 percentage points in specific tasks
- meaning the same model, on the same benchmark, can look completely different depending on how the questions are phrased. A benchmark score is a snapshot of one experimental setup. Your production workload is a different experimental setup entirely.
That's the thing evals can't fully escape: they measure a model under controlled conditions, but you deploy it into uncontrolled ones. The best practitioners treat evals not as a verdict but as an ongoing calibration - updating the test set as the workload evolves, tracking failure modes over time, and staying honest about what the number can and cannot tell them.