LLM Evals Measure More Than You Think - Until They Don't

Claude Opus 5 scores 97% on SWE-bench Verified. The same family of models, run on Scale AI's harder SWE-bench Pro under standardized scaffolding, scores closer to 59-80% depending on who's measuring. One benchmark, one model family, a gap of nearly 40 points. That gap is what evals are actually about - not the headline number, but what the measurement is really measuring.

Evals are the testing layer for AI systems. They tell you whether your model, your prompt, or your whole pipeline is doing what you think it's doing. Most engineering teams have a solid intuition for testing software: unit tests, integration tests, end-to-end checks. When an LLM enters the stack, that intuition gets you most of the way - but the floor changes under a few critical assumptions. Understanding where and why is the whole subject of this post.

What an eval actually is

An eval is a repeatable test that runs an input through your AI system, captures the output, and scores it against some definition of "good." That definition is where almost all the interesting engineering lives.

There are two fundamentally different ways to score:

Code-based assertions check whether the output satisfies an objective condition. Did the response return valid JSON? Did the agent call the right tool? Did the SQL query return the expected row count? Write a test, run it, get a pass or fail.

Code-based evals are cheaper to create and maintain than other kinds of evals. Since there is an expected output, you only have to run the LLM to generate the answer, followed by a simple assertion. This means you can run them on every commit to prevent regressions.

LLM-as-a-judge evals handle the questions code can't. Other questions don't have a clean answer in code. Did the chatbot actually address what the user was asking? Was the analysis grounded in the retrieved documents, or did the model fill gaps from its own training data? Was the tone right for a billing conversation? Code can verify form. It can't verify substance. LLM-as-a-judge evals were built for that gap.

LLM-as-judge calls take hundreds of milliseconds to a few seconds and can cost cents to dollars per call. The discipline of running heavy judges only on a small subset becomes load-bearing rather than aesthetic. In practice, good eval pipelines look like a pyramid: many cheap assertions at the bottom, a smaller set of LLM judge calls at the top.

How LLM-as-a-judge works inside

The mechanic is simpler than the name implies. An LLM-as-a-judge eval is a repeatable test that uses a separate language model to score an AI system's output against a rubric written in natural language. The judge model receives three inputs: the rubric (the criteria for what counts as good), the user's original input, and the system's response. It returns a score, a pass or fail verdict, or both.

Here is what that looks like in a concrete case. A support bot returns an answer to a billing question. The rubric says: (1) the answer must be grounded in the retrieved document, not the model's training data; (2) the tone must be professional; (3) it must not suggest action the policy doesn't support. The judging model reads the criteria, reads the output, and generates a score. You get a pass or fail per criterion, plus a short explanation - and you can diff that against a previous run to catch regressions.

An LLM judge agrees with human reviewers about 85% of the time - higher than two humans agree with each other on the same task. That's the bet behind LLM-as-a-Judge, now the default method for evaluating LLM applications at scale.

That 85% figure is the reason the method is now standard. It's also the reason it isn't sufficient on its own. LLM-as-a-judge faces several challenges, including misalignment with human judgments, biases in various forms, and inconsistencies in decision-making. The most documented bias is verbosity: judges consistently prefer longer answers, regardless of whether those answers are more correct. A judge prompt that doesn't explicitly penalize unnecessary length will drift toward rewarding verbosity over precision over time.

The fix most teams reach for first is rubric decomposition. Because LLM behaviors are multi-valid and non-deterministic, the field has shifted to an analytic rubric approach. Unlike holistic rubrics, which provide a single opaque score, analytic rubrics score criterion-by-criterion. This allows for regression root-cause analysis that a simple pass/fail check cannot provide. If a model's helpfulness improves but its safety tone regresses, analytic rubrics let you see exactly where the quality distribution shifted.

The benchmark trap - and what it tells you about your own evals

Public benchmarks are a version of the same problem, at scale. These benchmarks test a fixed distribution of tasks that may have little overlap with what your agent does in your environment. They're useful for model selection and for tracking progress over time. They are not a substitute for evals built against your own failure cases.

The saturation story on SWE-bench Verified is worth understanding in detail, because it shows exactly how an eval can degrade. Three of the 79 models evaluated reach 95% or better, and the leader, Claude Opus 5 at 97.00%, sits 3.00 percentage points from a perfect score - little room is left to separate frontier models on this benchmark alone. When the benchmark launched in late 2023, it was a genuine inflection point - for the first time, AI coding evaluation moved beyond toy problems and single-function completions to real GitHub issues from real open-source projects. Early top scores were around 20-30%.

Now consider what happened to the measurement validity. OpenAI's Frontier Evals team stopped reporting SWE-bench Verified in early 2026 after an internal audit of 138 problematic tasks found that more than 60% were unsolvable as written due to flawed tests - and that frontier models could reproduce the gold-patch solutions verbatim from just the task ID. Independent research found that 32.67% of successful SWE-bench Verified patches involved solution leakage, and that models recall the correct file paths from training data up to 76% of the time.

When a model can "solve" a third of the tasks partly by remembering the answer, a 90%-plus score is measuring memory as much as capability.

The less-told comparison is SWE-bench Pro, Scale AI's contamination-resistant alternative. The same frontier models clearing 80-95% on SWE-bench Verified solve only ~59% of tasks on SWE-bench Pro under standardized scaffolding. The gap is caused partly by contamination and partly by scaffold inflation - leaderboard entries are built on heavily tuned agent harnesses, and the harness alone can move a result 10-20 points.

This has a direct lesson for teams building their own evals. Progress that rapid is impressive and also a signal that saturation is approaching. Evals that differentiated models six months ago may no longer discriminate between good and great. Your golden dataset needs to grow with your failure cases - not stay frozen at the prompts you hand-wrote on day one.

How to build evals that don't rot

The hierarchy is simpler than most frameworks make it sound:

• Start with deterministic checks. Unit-test-style checks that run in milliseconds: does the output contain a phone number when it shouldn't? Does the classifier return one of the four expected labels? Does the JSON parse without errors? These are not quality evaluations - they're sanity checks. They catch obvious breakage before anything touches a judge.

• Build your rubric from real failures, not hypotheticals. Every criterion in your judge prompt should trace to a real production incident or a real user complaint. Generic rubrics produce generic scores.

• Validate your judge against human annotations. Make LLM judges more reliable by writing explicit evaluation steps, using strict mode for binary pass/fail checks, splitting complex logic into a decision tree, validating judge scores against human annotations, and inspecting score reasons during debugging.

• Treat your eval dataset as a living artifact. Add new cases after every incident. A static golden set is a snapshot; your system's failure modes are not static.

• Read public benchmarks as directional, not decisive. A model's SWE-bench Verified score in 2026 carries contamination risk and scaffold inflation that make head-to-head comparisons unreliable. Use it as a rough signal, not a deciding factor.

The honest position on evals is that they are themselves a measurement problem. A poorly designed eval gives you false confidence, which is worse than no eval at all - it just takes longer to discover.

LLM evals: common questions

What is an LLM eval?

An LLM eval is a repeatable test that scores an AI system's output against defined criteria. Evals split into two types: code-based assertions (deterministic, fast, cheap) and LLM-as-a-judge calls (probabilistic, slower, handles nuance). Together they cover both objective correctness and subjective quality.

How does LLM-as-a-judge work?

A separate language model receives three inputs - a rubric, the original user input, and the system's output - and returns a score or pass/fail verdict per criterion. The judge reads the rubric as a prompt, reasons against it, and generates structured feedback. Judge accuracy is calibrated by validating scores against human annotations.

Are LLM eval scores reliable?

LLM judges agree with human reviewers roughly 85% of the time, which meets or exceeds inter-human agreement. Reliability drops on tasks requiring deep domain expertise, numerical precision, or when the rubric is vague. Running two judges on the same output and checking agreement is a practical reliability sanity check.

Why do SWE-bench scores differ so much between providers?

Primarily scaffold inflation and contamination. SWE-bench Verified entries are mostly self-reported, and the agent harness used to run the benchmark can shift results by 10-20 points. Contamination research found that frontier models recall correct file paths from training data up to 76% of the time, meaning part of what's being measured is memorization.

What is the difference between evals and benchmarks?

Benchmarks are published, shared evaluation datasets used for cross-model comparison. Evals are tests you build and run against your own system. Benchmarks are useful for model selection; they are not a substitute for evals built against your specific failure cases and production distribution.