Pick any model leaderboard today and you will find MMLU contamination rates as high as 29%, with one study showing Mistral dropping 13 points when re-tested on questions it had never seen during training. The number on the scoreboard and the number your users experience are not the same number. Understanding why - and how evals are supposed to fix that - is one of the highest-leverage things a team building on top of LLMs can do.
What LLM evals actually measure
Evals are not a single thing. LLM evaluation measures whether this model does what your specific application needs it to do - correctness, faithfulness, relevance, and safety are four different dimensions, each requiring a separate metric. That framing matters because most teams reach for benchmarks when they should be asking a narrower question: does this model handle the inputs my users actually send?
The four dimensions worth tracking are distinct enough to fail independently:
Correctness - is the output factually right relative to a known answer? Measurable with exact match for structured outputs, ROUGE for extractive tasks, or an LLM judge comparing to a reference. Correctness matters most where wrong answers have concrete consequences: medical queries, financial data, code generation.
Faithfulness - does the output stay grounded in the provided context? This is the RAG-specific failure mode: a model that invents plausible but unsupported answers.
Relevance - does the response address what was actually asked, not just the topic?
Safety - does the output stay inside your application's policy constraints?
The three grading methods, and when each breaks
Three methods do the measuring: reference-based metrics for constrained outputs, LLM-as-a-judge for open-ended quality, and human evaluation to establish ground truth.
Reference-based metrics - BLEU, ROUGE, exact match - compare model output to a known correct answer. They measure n-gram overlap or string equality. Use them for classification, structured extraction, slot filling - any task where valid output variation is minimal. Do not use them for open-ended generation.
A summarizer that produces a demonstrably better summary than the reference answer gets penalized by ROUGE. The metric is doing exactly what it is designed to do. It is just the wrong tool for the job.
LLM-as-a-judge is where most production teams land for anything conversational. G-Eval is a flexible evaluation method that uses chain-of-thought prompting to break down and score model outputs based on user-defined criteria. You provide a prompt with a task description and evaluation guidelines, and the LLM generates step-by-step reasoning before assigning a score. This structure helps the LLM reflect more deeply on the evaluation task to improve consistency and alignment with human judgment.
Human evaluation is the ground truth anchor. LLM-as-a-judge scores against criteria that generic benchmarks cannot capture, but to ensure automated scores remain accurate, those scores need validation through human annotation. That validation confirms that the judge's assessments align with expert judgment from domain specialists.
| Method | Cost | Best for | Breaks when |
|---|---|---|---|
| Reference-based (ROUGE/BLEU/exact match) | Near zero | Structured outputs, classification | Multiple valid answers exist |
| LLM-as-a-judge | ~$0.002-0.02 per eval | Open-ended generation, RAG faithfulness | Judge has systematic style or length bias |
| Human evaluation | Highest | Ground truth, calibration | Scale; too slow for CI/CD |
The bias your eval judge is hiding from you
The LLM-as-a-judge literature spent years on position bias - whether the judge favors the first answer it sees. That problem appears to be mostly solved. Position bias is negligible. Contrary to the emphasis in prior work, position bias is ≤0.04 across all five tested models, likely reflecting improvements in instruction tuning since earlier studies.
The bias nobody talks about is style bias. Style bias is the dominant bias, measured at 0.76-0.92 across all models, far exceeding position bias (≤0.04), yet it has received minimal research attention. Concretely: your judge picks the response that reads like clean, structured prose over an equally accurate answer written in a different register - even when the content is identical.
Verbosity bias is real but splits along model lines. Some judges exhibit classical verbosity bias (favoring length regardless of content), while Claude and GPT-4o are more quality-sensitive, penalizing filler on expansion pairs while still rewarding genuine completeness on truncation pairs.
The practical fix: jury-of-judges setups - three independent judges, majority verdict - reduce all three biases and cost three times as much. Whether that tradeoff earns it depends on what you are evaluating. For a support triage bot routing 500 tickets a day, running three judges per production sample is probably not worth it. For a medical summarization tool, it likely is.
Why leaderboard scores are unreliable, and what to measure instead
Benchmark data contamination poses a major challenge in evaluating LLMs. Contaminated benchmarks can overestimate model performance by testing on data the model has already seen. This is not a marginal effect. Studies have shown that some models perform well on GSM8K but perform poorly on newly released Hungarian high school exams, and that changing the numbers in GSM8K questions can cause a performance drop. Widely used datasets including MATH and GSM8K exhibit score saturation, where models achieve near-perfect performance by memorizing correct answers during training, rendering the evaluation results unreliable.
Due to large-scale web scraping, newly released benchmarks are quickly copied across repositories, discussion forums, and derivative datasets, which makes it difficult to exclude them reliably from new training corpora. Even gated benchmarks may be indirectly incorporated through data aggregation pipelines, model distillation, or continual pretraining. This shortens the effective lifespan of benchmarks as clean evaluation tools.
The practical response is not to abandon benchmarks - they do give you a useful first signal when comparing models - but to supplement them with task-specific evals on your own data. A support team triaging 300 tickets a day will learn more from a 200-case golden dataset drawn from real customer queries than from any public leaderboard.
Running evals in CI/CD, not just pre-launch
Teams are moving beyond one-off evaluations to continuous validation that runs automatically with every deployment, giving them confidence that prompt changes, model swaps, and code updates do not degrade application quality.
The layered approach that actually scales:
Deterministic checks first. Engineering teams manage deterministic checks - JSON schema validation, regex patterns. These are quick, cost-effective, and can run on every request.
LLM-as-a-judge on a sample. Run judge evals on a sampled subset in CI - not every request. Eval suites that take over 10 minutes block your deploy pipeline. Token costs explode when you run evals on your full test suite with every commit.
Human spot-checks on production failures. Offline evals are necessary but not sufficient. Sample and score production traffic continuously.
A tool like Beagle, watching a support or engineering channel, can close the loop here: when a Slack thread surfaces a response that users flag as wrong, that thread becomes a new eval case - logged with context, ready to add to the test dataset.
Beagle in action#ai-support-ops, 2:47pmFor agent evals specifically, single-turn accuracy metrics miss most of what matters. Thread-level evaluation measures what actually matters for AI agent performance: task completion (whether the agent accomplished what was asked), user outcome (whether the conversation ended successfully), and agent trajectory (whether the agent took a reasonable path, including appropriate tool calls).
LLM evals: common questions
What is LLM-as-a-judge and is it reliable?
LLM-as-a-judge uses a second language model to score the outputs of your primary model, given a rubric. It correlates well with human judgment at scale but has known biases - style preference being the largest in current models. Calibrate it against human annotations on your specific task before trusting the scores.
How is LLM evaluation different from benchmarking?
Benchmarks test a model's raw capabilities on standardized tasks. Evals test whether a specific application, using a specific model and prompt, does what your users need. Benchmark scores are input for model selection; evals are ongoing quality signals for your deployed product.
What is benchmark contamination in LLMs?
Benchmark contamination happens when test questions appear in a model's training data, inflating scores through memorization rather than genuine reasoning. MMLU shows contamination rates around 29% in audit studies, and some models drop double digits in score when tested on genuinely unseen questions.
How many test cases do I need for a useful eval?
Start with 50 real production failure cases - queries where your current system gave a wrong or problematic answer. That beats a 1,000-case synthetic dataset every time. Grow the set incrementally as new failure modes surface in production.
Should evals block deployments?
Deterministic checks - schema validation, safety filters - should block. LLM-as-a-judge scores should gate: flag and require human sign-off when a metric drops more than a defined threshold. Running every judge eval synchronously in CI is too slow and expensive at any meaningful scale.