A support bot your team shipped last quarter answers a question about refund policy. The answer is mostly right, subtly wrong, and written in a tone that's technically correct but weirdly cold. No unit test catches this. No error is thrown. The model returned a 200 and moved on.
This is the problem evals exist to solve - and why so many teams who skip them end up doing what OpenAI's own documentation calls "vibe-based testing." Building with LLMs requires ongoing experimentation rather than "vibe-based" testing. Evals are the structured alternative: a repeatable way to measure whether a model is doing what you actually need it to do, not just what it's technically capable of.
What an eval actually is
An eval is a structured test that measures how well an LLM performs on a specific task. Each eval contains a data source with test cases, testing criteria that define correctness, and a prompt or model configuration to evaluate. The key difference from traditional software testing: unlike unit tests that check exact outputs, evals measure quality through programmatic checks or model-based grading.
Evaluations test model outputs to ensure they meet style and content criteria that you specify. Writing evals to understand how your LLM applications are performing against your expectations, especially when upgrading or trying new models, is an essential component to building reliable applications.
In practice, an eval has two components. First, a dataset: a collection of input/output pairs that represent the behavior you care about - happy paths, edge cases, the things that have failed before. Ensure your test data includes typical cases, edge cases, and adversarial cases. Second, a grader: the mechanism that decides whether any given model response passes or fails. Getting the grader right is where things get interesting.
The three grading methods
The way you grade model outputs defines how useful your eval actually is. There are three main approaches, each with clear tradeoffs.
Exact match is the simplest: the model's output must equal a known correct answer. For tasks such as multiple choice question answering where only a few choices are valid, exact match or likelihood are often used. If you're classifying support tickets into three buckets - Hardware, Software, or Other - exact match is fine. The answer is binary. But evaluating generations with more than a few tokens presents new challenges as there are often a large number of valid and correct answers. For shorter generations such as question answering, token overlap between the generated answer and the labeled answers are measured with F1, BLEU, ROUGE, and others. These token overlap metrics often struggle when a generated answer is correct but not in the list of labeled answers, as in the case with paraphrasing.
That struggle is not theoretical. If a user asks "when did Obama win his first presidential election?" and the model answers "Obama was first elected president in 2008," an exact-match grader looking for "2008" will fail it. The answer is right; the metric is wrong.
Statistical overlap metrics like BLEU and ROUGE count n-gram matches between a generated response and a reference. BLEU was originally designed for machine translation, ROUGE for summarization. They're useful, but limited: high scores don't guarantee quality.
Overlap-based metrics often don't correlate well with human judgements and aren't suitable for very open-ended tasks. Modern alternatives, such as embedding- or LLM-based evaluations, offer more context-aware assessments.
LLM-as-a-judge is the third method, and increasingly the dominant one for anything involving natural language. LLM-as-a-judge evaluation uses a language model as the evaluator for another language model's output. Instead of relying only on exact string matching, BLEU, ROUGE, or manual review, you give an LLM judge the interaction you want to evaluate and ask it to score the output against a specific criterion.
A practical rubric from Futurice's eval guide: pass the model's output to a judge prompt that asks for scores on empathy, accuracy, and completeness, then return structured JSON. The judge never sees whether the output "looks right" - it reasons against the rubric.
LLM judges typically operate in two modes: pointwise evaluation, where outputs are scored directly against criteria such as factuality and helpfulness, and pairwise comparison, where two outputs are compared and the preferred one is selected with justification.
Why LLM-as-a-judge is good, and where it breaks
Frontier models have been shown to approximate human judgments in structured evaluation tasks when provided with explicit criteria and grading rubrics. This paradigm, often referred to as "LLM-as-a-judge," relies on prompting models to assess outputs according to predefined dimensions, sometimes achieving substantial agreement with expert raters.
The practical wins are real: LLM-as-a-judge provides businesses with high efficiency by quickly assessing millions of outputs at a fraction of the expense of human review. It is adequate at evaluating technical content in situations where qualified reviewers are hard to come by, allows for continuous quality monitoring of AI systems, and produces repeatable results throughout evaluation cycles.
But the method has known failure modes. Using LLMs to evaluate quality can introduce multiple biases. Models are known to prefer certain styles of output due to their training, potentially biasing evaluation when the same model is used to generate output and evaluate. This is why for best results, it makes sense to use a different model to do grading from the one that did the completion, like using GPT-4 to grade GPT-3.5 answers.
There's a subtler risk too: the authors of G-Eval found that GPT-4 gave higher scores to AI summaries than to human ones, even when humans actually preferred the human-written versions. This suggests that LLMs might use the same internal standards for both writing and judging text - a bias that can lead to "self-reinforcement."
Building an eval that actually catches regressions
Evals are methods to measure and improve the ability of an AI system to meet expectations. Similar to product requirement documents, evals make fuzzy goals and abstract ideas specific and explicit.
The structure is straightforward. OpenAI's Evals API, for example, requires two ingredients:
a data_source_config - a schema for the test data you will use - and testing_criteria: the graders that determine if the model output is correct.
You can use string-match graders for deterministic outputs, code-execution graders for tasks involving structured data, and model-graded scores for open-ended text.
A concrete starting point: start with one critical path. Build 50 test cases. Pick two or three metrics. Run them before each deployment. That's it. The test suite grows from there, especially after production failures - each one becoming a new case that guards against repeats.
Set up continuous evaluation to run evals on every change, monitor your app to identify new cases of nondeterminism, and grow the eval set over time. Some teams wire this directly into CI/CD. A teammate like Beagle, running inside Slack or Teams, produces exactly the kind of natural-language output that benefits from this kind of pipeline: did the summary correctly reflect the meeting decisions, and was the tone appropriate for the channel it was posted in?
The distinction worth holding onto throughout: a benchmark like MMLU or HumanEval is a standardized test used to compare models in isolation. An eval is a more specific term for the structured tests you implement to measure how well an LLM application performs a specific job for your unique use case. Public benchmarks tell you what a model can do. Evals tell you whether it's doing your thing, correctly, reliably, every time you ship a change.
That definition is the hard part. The infrastructure is almost secondary. Once you know what good looks like, you can measure toward it. Without that, you're back to vibes.