Someone on an AI product team tweaks a system prompt on a Friday. They test it on three queries, the answers look sharper, and they ship. The following Monday, customer support starts getting complaints that the assistant is now too terse in refund cases - polite but cold. Nobody noticed because nobody measured it. The change passed the vibe check and failed the real one.
This is the problem evals solve. Not glamorously, but reliably.
What an eval actually is
An eval is a test that scores a language model's output against a defined criterion. That criterion can be simple - does the response contain the word "refund"? - or subjective: is this response appropriately empathetic? The mechanics differ, but the structure is always the same: input, output, score.
Evaluating an LLM is fundamentally different from evaluating a classifier. There is no single accuracy score. Good responses depend on intent, style, and context. A response can be grammatically perfect and completely wrong.
That's why early evaluation borrowed metrics from older NLP work. Automated benchmarks like MMLU measure general capability using multiple-choice questions - fast and reproducible, but disconnected from any specific use case. Task-specific metrics like ROUGE and BLEU measure output quality against reference answers - reliable for structured tasks, but useless for open-ended ones. Neither is much help if you are building a customer-facing assistant and need to know whether a rephrased prompt made it worse at handling angry users.
Three kinds of evaluation
It helps to separate them clearly.
Deterministic evals check hard facts: did the model call the right tool, return valid JSON, stay within a word limit, avoid a banned phrase? These are fast, cheap, and never wrong. Run them first.
Reference-based evals compare the model's output against a "golden" answer - a human-approved response to the same input. This is an offline evaluation where the model compares a response to a golden reference. It works well during the experimental phase and for regression testing after updates to the model or prompt. For example, in a Q&A system, the judge checks whether the new response is similar to the previously approved answer.
LLM-as-a-judge is where the field has mostly landed for anything subjective. You send a prompt to a separate model - often a frontier model like GPT-4 or Claude - with a rubric, and it scores the output. The idea is simple: ask an LLM to judge the text output using guidelines you define. You can ask it to evaluate things like politeness, bias, or tone.
The judge is not the model you are evaluating. It is a separate model, given a scoring rubric, asked to act as a strict reviewer.
Why teams use LLM-as-a-judge at scale
The math is what makes it practical. Evaluating a thousand model responses with human reviewers can take days or weeks and cost thousands of dollars. An LLM judge does it in minutes for a fraction of the cost. That efficiency gain means you can run evaluations on every pull request, every prompt change, every model swap - turning evaluation from a periodic audit into continuous regression testing.
There is also a readability advantage over older numeric metrics. Unlike BLEU, which returns an opaque float, an LLM judge can articulate why it scored a response the way it did. If a response gets penalized, the judge can point to the specific claim that contradicted the context, or which part of the user query went unaddressed. This transforms evaluation from a measurement step into a debugging tool.
Back to the Friday prompt edit: if the team had been running a judge scorer on tone and empathy, the regression would have surfaced in the eval run before the change merged. If a factuality scorer requires 85% or higher and a prompt change reduces it to 78%, the gate prevents the change from reaching production.
Offline vs. online: both matter
There is a meaningful difference between evals you run before shipping and evals you run after.
Offline evaluation runs against a curated dataset, typically pre-merge in CI or on a schedule. It is deterministic and reproducible. Online evaluation runs against live production traffic, scoring outputs as they are emitted. It captures real-world drift, distribution shift, and rare failure modes that no curated dataset can predict.
A curated dataset is something you build deliberately: 50 to 200 representative inputs covering common cases, edge cases, and known failure modes. Even though standardized benchmarks are useful, it is important to create specific test scenarios that are as close as possible to the real application. These case-specific scenarios enable evaluation in the conditions the model will actually encounter.
The dataset is not a one-time artifact. Feedback loops between production monitoring and the evaluation dataset ensure continuous expansion of coverage. When a user flags a bad response or a production scorer detects a low-quality output, that interaction becomes a new test case in the dataset. Over time, the dataset grows to cover real-world failure modes that initial development could not have anticipated.
A teammate like Beagle - living inside Slack where users surface frustrations in real time - is exactly the kind of system that benefits from this loop: flagged responses in a channel feed directly back into the dataset the eval suite runs against.
The calibration problem
LLM judges have a known weakness: they are probabilistic, and their scores drift unless you anchor them to human ground truth.
Teams that collect human corrections, build few-shot examples, and track agreement metrics get automated scores they can use to make shipping decisions with confidence. Early on, developers wrote rubrics and configured LLM judges hoping the scores would hold, but they did not. Running evaluators in production has taught us that the evaluator prompt directly determines trustworthiness. Getting the prompt right requires systematic alignment to human preferences through deliberate iteration, not intuition.
The fix is straightforward, if unsexy: periodically sample a few hundred scored outputs, have a human review them, note where the judge disagrees with the human, and adjust the rubric. Running judges in production helps build a data flywheel. Production traces feed the observability layer, which surfaces usage pattern insights. Those insights inform datasets. Datasets power evaluations. Evaluations drive improvements. Improvements generate better traces, and the cycle continues.
What this means in practice
Evals are not a research concern. LLM outputs are non-deterministic, multi-step, and easy to break with a vendor model swap or a prompt edit. Evaluation is the mechanism that catches regressions before users do.
The invisible problem with that Friday prompt change was not that someone made a mistake. It is that the team had no fast feedback loop between "we changed something" and "here is what got worse." Evals are that loop. They do not guarantee good outputs. They guarantee that you notice quickly when the outputs get bad.
That is a quieter promise than most AI tooling makes. It is also the one that actually holds.