How LLM Evals Work, From Golden Dataset to Judge Score

A refund agent at a fintech company scored 0.89 on helpfulness. The eval suite was green. Three weeks later, a user got a refund quote off by an order of magnitude, and support tickets spiked. The JSON schema for the tool call had drifted quietly across three releases. The judge model read the prose around the broken field and called it fine.

That is what an LLM eval looks like when one layer is missing. And it illustrates the whole problem: unlike a unit test, an LLM eval can pass on the wrong thing.

What an LLM eval actually is

An LLM eval is a structured test that scores what a model or agent produces against a defined expectation - whether that's a reference answer, a policy rule, a functional check, or a quality rubric. That definition sounds simple. The machinery underneath it is not.

Every eval run has three components: a dataset of inputs, a method of generating outputs, and a grader that decides whether each output passes. Change any one of those three, and you change what the eval measures. Most teams learn this by discovering their eval suite is green while production is broken - usually because the dataset does not cover real traffic, or the grader is measuring the wrong thing.

During testing, before deployment, evals compare versions so you can see whether a change made the agent better or worse. During monitoring, after deployment, evals score real production traces and feed failures back into the next version. That feedback loop - not the score itself - is what makes evals useful.

Agents can state false information as fact, be manipulated into ignoring their own instructions, produce harmful content, or call their tools in the wrong order. None of these failures show up as crashes or error messages. Code can run cleanly and pass functional tests, yet still return an answer that's wrong, unsafe, or against the rules.

The two grader types and when each one breaks

There are two main ways to evaluate model completions: writing validation logic in code, or using the model itself to inspect the answer. Neither is strictly better - they fail in opposite directions.

Deterministic graders (exact match, regex, JSON schema, includes) are cheap and fast enough to run on every output and every failure has a clear, auditable reason, with near-zero latency overhead under 10ms. Their weakness: they are brittle - they cannot understand intent or semantic meaning. If a model says "The sun is rising" instead of "The sun is coming up," a strict deterministic check might flag it as a mismatch.

Model-graded evals (LLM-as-a-judge) leverage one model as a judge to evaluate responses generated by another model based on specified criteria. This approach addresses evaluation challenges for creative, subjective, or complex tasks where ground truth can be ambiguous or multifaceted. Their weakness is cost and bias - which the research section below covers specifically.

The practical pattern that has emerged is a pyramid: run deterministic checks first on every output, then route only the cases that pass structural validation to a model judge. The deterministic layer filters out 30 to 60 percent of failures - broken JSON, missing citations, malformed tool calls, banned phrases - cheaply enough to run on every request, so judge tokens can go to the cases that actually need reasoning.

The fintech refund example at the top of this post was a schema failure. A parser would have caught it in under a millisecond. The judge read the surrounding prose, scored it 0.89 helpful, and shipped the bug.

What a golden dataset is and how big it needs to be

A golden dataset is a curated, versioned collection of test cases - inputs, context, and expected behavior - that you evaluate your LLM system against. It is your trusted reference: hand-checked and representative, so you can compare runs over time and detect regressions. It is the single most important asset in an evaluation practice, more decisive than the specific metrics or framework you choose.

Size guidance from practitioners is fairly consistent:

- 20-50 test cases for a quick sanity check during development - enough to catch obvious regressions, but not enough for statistical confidence. Around 100 test cases for reliable metric reporting at a single threshold.

- 200-500 examples for a production-ready suite covering major use cases and edge cases; 1,000+ for a mature system with regular additions from production failures.

- Databricks used 100 questions from internal documents in one evaluation, while another used 1,000 QA pairs, each validated three times by two labelers. Microsoft's Copilot Teams recommends 150 question-answer pairs for complex or broad domains.

The non-obvious point: grow the dataset deliberately from real failures rather than padding it with synthetic cases. Sixty distinct, meaningful cases beat five hundred random ones.

A golden dataset is the fixture your CI eval gate runs against on every build, and the gate is only as honest as the fixture under it. A serious version has four buckets: a stratified sample of production traffic, an adversarial library, deliberately constructed edge cases, and replays of failures that already shipped.

The real problem with LLM-as-a-judge

LLM-as-a-judge is the right tool for subjective quality - tone, factual faithfulness, task completion. It is the wrong tool when you need a reproducible, auditable verdict. The judge is probabilistic: it gives a different verdict on re-run, carries measurable bias, and can't tell you why it passed an output.

The bias research is now specific enough to act on. A 2026 study across five judge models from Google, Anthropic, OpenAI, and Meta found that style bias is the dominant bias, measuring 0.76-0.92 across all models, far exceeding position bias at ≤0.04 - yet it has received minimal research attention.

When an LLM judge evaluates two responses, the longer one tends to win, even when the extra length is padding, redundancy, or restating the same point in different words. The bias is strong enough that researchers can inflate scores by simply adding a summary paragraph that repeats the response's key points.

Padding a response with redundant detail can shift scores by 0.5 to 1.5 points on a 5-point scale.

There is also a self-preference problem. GPT-4 favors itself with a 10% higher win rate and Claude-v1 with a 25% higher win rate respectively

• which means if you use the same family of model as your judge that you used to generate responses, your scores are systematically optimistic. Using a judge from a different provider than your subject model is a cheap mitigation most teams skip.

Stronger judges are better but not immune. Even the strongest judges still change 14.7% of verdicts under A/B reversal, so position randomization and slice-level bias reporting remain necessary even when using high-accuracy judges.

One practical implication most coverage misses: anything weaker than GPT-4 produces noisy scores. Compose graders for layered evaluation - a deterministic check first, then a model-graded fallback. This matters for cost, too. A judge model call at 300ms and a fraction of a cent sounds cheap until you multiply by a million traces a day. The first month can land at $40K.

The non-obvious second-order point: if your eval pipeline rewards verbosity, your team will learn - gradually, without realizing it - to write prompts that produce longer outputs. The eval pipeline itself shapes the behavior of the system it is measuring. That feedback loop runs in both directions.

How LLM evals: common questions

What is the difference between an eval and a benchmark?

A benchmark is a fixed, public test suite measuring general capability - MMLU, HumanEval, MATH. An LLM eval is a custom test suite measuring your specific application against your specific requirements. Generic benchmarks do not predict how your LLM app performs in production. The gap between benchmark scores and real-world performance is where custom evaluation frameworks live, and it is where most teams lose weeks before figuring it out.

How many test cases does a golden dataset need?

Start with 50 to 100 well-chosen examples for a single feature, split roughly into 40 to 60 core cases and 15 to 25 edge cases. This is enough to catch most regressions without being slow or expensive to run. Grow it from real production failures rather than synthetic padding. Quality of coverage matters far more than raw count.

When should I use LLM-as-a-judge vs. deterministic checks?

Deterministic checks should handle everything that can be measured directly - format, schema compliance, and required fields - as these checks are inexpensive and predictable. LLM-as-a-judge should focus only on subjective dimensions that require language understanding. Separating the two improves reliability and reduces the impact of judge errors.

Can the same model judge its own outputs?

Technically yes, practically risky. Judges tend to assign more favorable evaluations to their own outputs. Evaluator impartiality remains fragile when the judge is also part of the models being evaluated. Use a judge from a different provider than the model generating responses when self-preference bias would distort your results.

What does it mean when eval scores stay flat but quality feels worse?

Your eval suite can become a comfortable lie where everything passes but quality is actually declining. One signal to watch: if your eval pass rate stays suspiciously flat while user complaints increase, your evals have drifted. The test cases no longer represent real usage, and the gate reads green because nothing in the suite fails - not because nothing is failing.