LLM-as-Judge: How Automated Eval Actually Works

Annotating 1,000 model outputs with human crowdworkers costs around $300 and takes days. AlpacaFarm - Stanford's NeurIPS 2023 work - showed the same job costs $6 using an LLM judge, 50x cheaper, with high agreement. That number is why every serious AI team now runs some form of LLM-as-judge eval. The question is whether they're doing it in a way that actually tells them something true.

What an LLM judge actually does

It works by feeding an evaluation prompt - containing the criterion, the input, and the generated output - to a judge LLM, which returns a numeric score, a binary verdict, or a preference between two outputs. That's the whole mechanism. No magic.

The inputs a judge receives look like this:

• The original prompt - what the user or system asked
• The candidate output - the response under evaluation
• A reference output (optional) - a ground-truth answer to compare against
• A rubric - the criterion: helpfulness, factual accuracy, format compliance, whatever "good" means for your use case

Score formats include integer or float on a scale, binary pass/fail, ordinal verdict (A wins / tie / B wins), or a multi-dimension JSON object covering helpfulness, harmlessness, and factuality.

The key insight is that the rubric is doing most of the work. A vague rubric ("is this a good response?") produces noisy, nearly-useless scores. A rubric that says "does this response cite a specific figure from the retrieved document, without inventing numbers?" produces tight, reliable signal. Writing the rubric is harder than wiring up the API call.

Pairwise comparison vs. direct scoring

There are two main judge configurations, and they behave differently enough that the choice matters.

G-Eval (Liu et al., Microsoft, EMNLP 2023) showed that GPT-4, prompted with chain-of-thought reasoning and a structured "form-filling" scoring template, correlated with human judgment far better than any n-gram metric - 0.514 Spearman correlation on summarization, a genuinely large jump. That's the result that moved teams from BLEU and ROUGE toward LLM judges for open-ended tasks.

Manual evaluation of 100,000 responses takes 50-plus days. Traditional metrics like BLEU and ROUGE miss what matters: coherence, helpfulness, factual accuracy. LLM judges solve both problems simultaneously - which is why an LLM judge agrees with human reviewers about 85% of the time - higher than two humans agree with each other on the same task. That's the bet behind LLM-as-judge, now the default method for evaluating LLM applications at scale.

The three biases that break your scores

LLM judges are not neutral. The ScienceDirect 2025 survey and the EMNLP 2024 bias study document systematic biases that distort scores in predictable directions. If you do not measure these biases, your judge scores are unreliable.

The three that matter most in production:

Position bias. The tendency of judges to favor a response based on its order rather than its quality is one of the most extensively studied problems in LLM evaluation. In a pairwise comparison, the answer in the "A" slot gets a structural advantage. Fix: run both orderings (A vs B, then B vs A) and average.

Verbosity bias. Judges prefer a semantically equivalent but more verbose response over the original one. MLLMs are even more vulnerable to verbosity bias than to position bias. A model that writes 400-word answers to questions deserving 80 words will outscore a more precise competitor on raw judge ratings. AlpacaEval 2.0 addressed this directly by fitting a logistic regression to control for length differences before computing win rates.

Self-enhancement bias. Judges favor responses generated by themselves over those produced by other models. Self-enhancement bias is present, with judges tending to assign more favorable evaluations to their own outputs. Evaluator impartiality remains fragile when the judge is also part of the models being evaluated.

This last one has a concrete operational consequence most teams ignore: if you generate outputs with GPT-4o and evaluate them with GPT-4o, you are running a biased experiment. Pick pairwise comparison with both orderings, calibrate against human labels, and never use the same model family as generator and judge. Using Claude to judge GPT outputs, or an open-weight model to judge both, sidesteps this cleanly.

How production teams calibrate a judge they can trust

A judge you haven't calibrated is a random number generator with extra steps. Production teams now compute Cohen's kappa between judge and a labeled human sample before they ship any new rubric, and re-sample monthly to catch judge drift.

The practical loop looks like this:

1. Write a rubric around one specific criterion. Don't ask a judge to score helpfulness, accuracy, and format in a single prompt. Run three separate judge calls and aggregate scores.
2. Label a calibration set. 50-100 human-labeled examples is enough to check whether your judge tracks human judgment. If Cohen's kappa is below 0.6, rewrite the rubric.
3. Choose the judge model deliberately. Flash-tier judges (mini-class, Haiku-tier, Flash-tier) are substantially cheaper, fast enough for span-attached production scoring. Calibrate them against a frontier judge and labeled samples on your own data. Reserve frontier models for high-stakes CI gates.
4. For pairwise, swap the order. Run every comparison in both orderings. Only count consistent verdicts as confident signal.
5. Re-calibrate after model updates. Humans label 5 to 10 percent of the LLM output, focused on the lowest-confidence cases. Re-sample monthly. A fresh 50-example calibration set detects judge drift.

The model that Meta used to train Llama 3 applied this pattern internally: model release evaluation, with model-graded scoring (correctness, informativeness) running alongside human eval during post-training.

The 2026 production pattern is hybrid: metric-based for what is measurable, LLM-judge for what requires reasoning, human review for the failing 1-5% of samples flagged by either. Automated metrics (exact match, ROUGE) still earn their place for tasks where there is a deterministic right answer. The judge earns its place when the question is "was this actually a good response?" - which is most of what teams care about.

A teammate like Beagle fits into this loop naturally when the question comes up in a channel: pull a sample, run the comparison, surface the delta. The draft-and-approve model means a human still reads the summary before it influences a decision.

LLM-as-judge evals: common questions

What is LLM-as-judge evaluation?

LLM-as-judge evaluation uses one language model to score or compare the outputs of another against a rubric you define. You send the judge a prompt containing the input, the candidate output, and your criterion. The judge returns a score, a pass/fail verdict, or a pairwise preference. It replaces human annotation for high-volume, open-ended tasks where string-matching metrics fail.

How accurate is an LLM judge compared to human raters?

The decisive question is how well LLM judge scores agree with human judgment. Practitioner analysis and the arXiv bias study converge: on well-defined tasks (factual accuracy, format compliance, code correctness), judge-human agreement reaches 70-85%. On subjective tasks - writing quality, tone - agreement is lower. Calibrate with a labeled human sample before trusting raw scores.

What is position bias in LLM judges?

Position bias is an LLM judge's tendency to favor whichever response appears first in a pairwise prompt, independent of quality. The fix is to run each comparison twice with the order swapped and only count verdicts that are consistent in both directions. Most evaluation frameworks (DeepEval, Ragas, Langfuse) support this automatically.

Should you use the same model as generator and judge?

No. Judges tend to favor responses generated by themselves - self-enhancement bias is present, with judges assigning more favorable evaluations to their own outputs. Use a model from a different provider or family as your judge, or use an open-weight model you can audit. This is especially important in automated training loops where biased scores become training signal.

How many human labels do you need to calibrate an LLM judge?

Fifty to 100 carefully labeled examples is enough to detect whether a rubric is tracking human judgment. Compute Cohen's kappa between judge scores and human labels. Below 0.6 means the rubric needs rewriting. A fresh 50-example calibration set detects judge drift

• run one after every significant rubric change or model update.