LLM Evals: What Runs After You Edit a Prompt

Someone on your AI team rewrites one sentence in the system prompt. The output they were fixing looks better. They ship it. Three other cases that worked last week now quietly fail - wrong tool called, answer hallucinated, tone shifted - and nobody knows until a user complains. There was no diff to review, no test that turned red, no signal at all.

That gap is what LLM evals are built to close.

What LLM evals actually are

LLM evaluations - "evals" - assess the performance of a large language model to ensure outputs are accurate, safe, and aligned with user needs. That's the textbook definition. The practical one is sharper: evals are a test suite for behavior that can't be caught by assert output == expected, because LLMs are non-deterministic - the same input can produce different outputs, which makes exact-match testing insufficient.

The solution combines deterministic tests, reference-based scoring, LLM-as-a-judge, human review, and production signals to measure behaviors that do not have one exact correct output. These aren't competing approaches - mature pipelines use all of them at different points in the development cycle.

The three layers stack like this:

They gate the LLM-judge calls so you do not waste tokens on outputs that are obviously broken.

How a golden dataset and CI gate work together

The mechanism is simple enough to sketch in a paragraph. You maintain a golden dataset - a set of input/output pairs that represent known-good behavior for your application. This dataset lives in version control alongside your prompts. When someone opens a pull request that touches a prompt, the CI pipeline runs the new prompt against the golden dataset, computes quality metrics, and if eval scores drop below your defined thresholds, the PR fails.

Running a 500-sample eval dataset against a hosted model API can cost $5 to $50 per CI run, depending on the model and the complexity of the evaluation. That adds up at high PR velocity, so most teams split the cost: run a smaller smoke set of 50 samples on every PR, and run the full eval set only on merges to main.

The dataset itself needs maintenance. The strongest eval pipelines feed real production failures back into the golden dataset - when a user reports a bad answer, that case becomes a permanent regression test, so the same mistake can never silently return.

But here's the limit everyone forgets: a passing gate means "no worse than baseline on the cases we thought to test" - not "correct." CI evals catch regressions; they don't discover problems you never wrote a case for.

LLM-as-a-judge: how it scores outputs at scale

This is the part that sounds circular until you understand the mechanism. LLM-as-a-judge is an evaluation methodology where the judge model is given the input, the application's output, and a scoring rubric, then produces a score with reasoning. You're not asking the model if it's good - you're asking it to apply a specific, decomposed rubric.

Three primary patterns exist: pairwise comparison (the judge sees two outputs and picks the better one, which is more reliable than absolute scoring), direct scoring (the judge rates a single output on a rubric, typically 1-5), and reference-free evaluation (the judge assesses intrinsic qualities like consistency with source documents without needing a golden answer).

How accurate is this? Research on GPT-4 as a judge shows up to ~85% agreement with human annotators - higher than the ~81% agreement humans show with each other on the same tasks. But the accuracy is rubric-dependent. A judge given "rate helpfulness 1-5" reproduces the same ambiguity that wrecks human agreement; a judge given decomposed binary criteria with concrete anchors behaves much more consistently.

The failure modes are documented and real. Judges show position bias of up to 75% preference for whichever response appears first, verbosity bias toward longer answers, and self-enhancement bias - GPT-4 favoured its own outputs with about a 10% higher win rate. The fix is calibration: human-label a reference set, measure judge agreement on it, and re-run that check periodically as models update under the same API name.

Agent evals: why the final answer isn't enough

Plain LLM evals score a response. Agent evals score a run. While LLM evaluation assesses individual responses, agent evaluation must assess the complete trajectory: how agents reason about tasks, which tools they select, how they handle errors, and whether they achieve their goals efficiently.

An agent may select tools, plan across several steps, retrieve information, update state, recover from failures, and interact with a user over a complete session. Evaluating only the final answer can miss a wrong tool call, an invented parameter, or a failed action hidden behind a surface-level correct output.

Evaluate at three levels: end-to-end (did the task succeed?), trajectory-level (was the path efficient and sound?), and component-level (which retriever, tool, or sub-agent broke?). The component level is where regressions hide. A tool-selection accuracy drop from 94% to 87% looks small until you realize 1 in 13 agent runs is now choosing the wrong tool entirely - and the final answer might still look plausible.

The rule of thumb: use deterministic metrics for exact checks like tool correctness, and use LLM-as-a-judge for criteria that require judgment, context, or the agent's actual output. Tracing is the backbone - it shows where a metric failed, surfaces new failure modes you don't have metrics for yet, and paired with periodic human review keeps your evals calibrated as the agent drifts.

A newer approach - Agent-as-a-Judge - goes further. Instead of a single LLM rating a final answer, a judge agent evaluates agent performance on each sub-requirement and the overall task, producing detailed scores, while preserving the cost-effectiveness of LLM-based evaluation.

One study reported that Agent-as-a-Judge dramatically outperformed a standard LLM-as-a-judge that only saw final outputs - in one comparison, the agent judge's decisions differed from the human majority vote only 0.3% of the time, whereas a single LLM judge disagreed 31% of the time.

LLM evals: common questions

What is LLM evaluation and why does it matter?

LLM evaluation (evals) is the practice of systematically measuring whether a language model or AI agent produces correct, safe, and useful outputs across a representative dataset. It matters because LLMs are non-deterministic - the same prompt can produce different outputs, and a prompt change that fixes one case can silently break three others. Without evals, regressions reach users.

What is LLM-as-a-judge?

LLM-as-a-judge is an evaluation method where a capable model is given a rubric and asked to score another model's output. It achieves roughly 80-85% agreement with human annotators at a fraction of the cost and time. The main risks are position bias and verbosity bias, which calibration against a human-labeled reference set can largely correct.

How are agent evals different from LLM evals?

Agent evals must score the full execution trajectory - which tools were called, in what order, with what parameters - not just the final answer. A final answer can look correct while hiding a wrong tool call or an invented parameter earlier in the run. Component-level and trajectory-level metrics catch what end-to-end scoring misses.

Should evals run on every pull request?

Run a small smoke set (50-100 cases) on every PR to catch obvious regressions cheaply. Run the full dataset only on merges to main, where the cost is justified. Feed every production failure back into the golden dataset - that's how the gate gets sharper over time without someone maintaining it manually.

What tools do teams use to run LLM evals?

Common open-source options include DeepEval, promptfoo, and Evidently. Observability-integrated options include Langfuse and Arize Phoenix. Most now support LLM-as-a-judge workflows and OpenTelemetry-compatible tracing so eval spans attach directly to production traces.