GPT-4 can guess a missing answer option from MMLU questions at a 57% exact-match rate - not because it reasoned through the problem, but because the question was likely in its training data. That single finding, from a 2023 ACL paper, tells you more about how LLM evals work than any leaderboard ever will.
Understanding how LLM evals work means understanding three things: what runs, what gets scored, and why the score can be misleading even when the eval is working exactly as designed.
What an eval pipeline actually does
LLM evaluation has three distinct components: a test dataset - a curated collection of representative inputs covering typical cases, edge cases, and adversarial inputs; a success criterion - an explicit definition of what correct output looks like; and an evaluator - the mechanism that scores outputs against those criteria, which can be a script, a human reviewer, or another AI model.
Automated LLM evaluation workflows have two parts. First, you need data: synthetic examples, curated test cases, or real logs from your app. Second, you need a scoring method - which might return a pass/fail, a label, or a numerical score.
Run it in sequence and you get: feed the test inputs → generate responses → compare to a reference or rubric → compute a score. The part that trips people up is step three. "Compare to a reference" sounds simple. It isn't.
The three ways to score a response - and when each one breaks
Statistical scorers compare model outputs to a reference using surface-level patterns. Common examples include BLEU, ROUGE, METEOR, and edit distance. ROUGE measures how much of the reference content appears in the model response. These methods work best for tasks with clear references, such as translation or short summaries. However, they struggle with generative outputs where wording varies widely while still remaining correct.
Exact match - checking whether the response is identical to what's expected - is conceptually obvious but often too rigid. In open-ended scenarios, different wording can convey the same meaning.
That's why LLM-as-judge has become the dominant approach for anything open-ended. The core idea: present an LLM with the input, the application's output, and a scoring rubric, then ask it to evaluate the output. The judge model produces a score along with reasoning explaining its assessment.
This approach has become one of the most popular methods for evaluating LLM applications because it combines the nuance of human judgment with the scalability of automated evaluation.
The calibration figure people cite: LLM-as-a-judge uses a capable model to score outputs against defined criteria. OpenAI's evaluation documentation confirms that strong LLM judges achieve agreement rates of over 80% with human annotators.
What that figure doesn't show is the bias structure underneath it.
The bias problem inside LLM-as-judge
Research has identified position bias, where specific positions within a prompt are preferentially selected, and verbosity bias, which favors longer responses.
GPT-4, in particular, exhibits high self-preference bias - a finding that suggests GPT-4 as a judge may inadvertently reinforce its own style and policies.
The magnitude is not small. On ArenaHard, LLMs exhibit self-preferential bias: some over-rate their own answers relative to other judges, others under-rate them. On ArenaHard, this bias ranges from -38% to +90%.
The standard fix - prompting the model multiple times with different candidate orderings and aggregating results via majority voting - significantly increases computational costs. So in practice, most teams don't do it.
How public benchmarks actually score: MMLU as a worked example
Massive Multitask Language Understanding (MMLU) evaluates LLMs across 57 subjects, including elementary mathematics, US history, computer science, and law. The dataset contains over 15,000 multi-choice tasks from high school to expert level. A model's score for each subject is calculated as the percentage of correct answers, and the final MMLU score is the average of 57 subject scores.
Scoring is exact match on a letter choice. The model's score is calculated by determining the proportion of multiple-choice questions for which the model produces the precise correct letter answer. No partial credit. No rubric. The scorer is a string comparison.
That's also why HellaSwag works the same way: HellaSwag evaluates the common-sense reasoning capabilities of LLMs through sentence completion, testing whether models can select the appropriate ending from a set of four choices across 10,000 sentences.
Like MMLU, HellaSwag has saturated for frontier models at 95%+.
Leaderboard scores answer a single question: how did this base model perform on a fixed task set under controlled conditions? That answer helps narrow your model shortlist. It says very little once you add a retrieval layer, tool calls, a system prompt, user-specific context, and the actual workflow your agent has to complete.
The contamination problem is structural, not accidental
This is the part most benchmark discussions skip.
Johns Hopkins researchers cross-checked several models and benchmarks to estimate which items had been exposed to training data. On MMLU, roughly 29.1% showed contamination signals; on the Chinese benchmark C-Eval, 45.8% did.
Meta's own Llama 2 report found that 16% of MMLU items overlapped with its training data, some of them severely - with more than 80% of tokens matching.
The filtering labs do isn't airtight. GPT-3 cut anything matching 13 consecutive words, and GPT-4 raised the bar to 40-grams. But filters only fire when the text matches exactly. Reword a sentence slightly, translate it, or reshape a table, and the same problem becomes a different string that slips past the filter.
The consequence is a number that looks like reasoning but is partly recall. Standard benchmarks like MMLU and HellaSwag have been in the training data of most large models. High scores might reflect memorisation, not capability.
Unlike MMLU, BIG-Bench Hard requires chain-of-thought reasoning to score well, which makes it more predictive of real-world reasoning performance for models below the frontier tier. That's one reason teams doing serious model selection increasingly run BBH alongside MMLU rather than instead of it.
Beagle in action#ai-team, 3:02pmWhen to use each eval method
The decision isn't which method is best - it's which method fits the task. A practical decision rule: start with automated benchmarks to catch regressions; add task-specific metrics if your task is structured; use LLM-as-judge for instruction-following quality; reserve human eval for final validation before production.
Most fine-tuning evaluations need at least two types: one automated benchmark to catch capability regressions, and one task-specific or LLM-as-judge evaluation to measure actual task improvement.
| Method | Scoring | Good for | Breaks on |
|---|---|---|---|
| Exact match | String comparison | Classification, short answers | Open-ended output |
| ROUGE / BLEU | Token overlap | Summarization, translation | Paraphrased correct answers |
| LLM-as-judge | Rubric + model reasoning | Instruction following, tone, nuance | Self-preference bias, verbosity |
| Human annotation | Direct review | Ground-truth calibration | Scale, cost, speed |
| Public benchmarks | Avg % correct (multiple choice) | Model shortlisting | App-specific tasks, contamination |
A teammate like Beagle watching a production channel can flag when outputs diverge from the rubric a team has defined - a lightweight form of continuous eval that doesn't require running a full pipeline on every response.
How LLM evals work: common questions
What does an LLM eval actually measure?
An LLM eval measures how often a model's output meets a defined success criterion on a fixed set of test inputs. What that criterion is - exact string match, token overlap, human preference, or judge-model score - determines what the number actually means. No single eval covers everything.
What is LLM-as-a-judge and how does it work?
LLM-as-a-judge presents a capable model with an input, the system output, and a scoring rubric, then asks it to score and explain the result. It scales better than human review and handles open-ended outputs well, but carries known biases: self-preference, position sensitivity, and a tendency to favor longer responses.
Why do MMLU scores keep rising if the benchmark is hard?
Two reasons: models keep improving, and contamination inflates scores over time as benchmark questions appear in training data. Johns Hopkins researchers found roughly 29% of MMLU items showing contamination signals across common models. High frontier scores on MMLU now partly reflect recall, not just reasoning.
How is an application-specific eval different from a public benchmark?
Public benchmarks test a base model on generic multiple-choice tasks under controlled conditions. An application-specific eval tests your actual system - with its system prompt, retrieval layer, tools, and real user inputs - on your definition of a correct answer. The two measure different things.
When should teams run evals in production, not just before launch?
Whenever the input distribution can shift after deployment. Real traffic exposes edge cases, adversarial inputs, and prompt drift that pre-launch test sets miss. Sampling production traffic and scoring it continuously - even at low volume - catches regressions before users do.