In a previous post I listed eleven ways an agent eval can lie, and one of them was trusting an uncalibrated LLM judge. This post covers what can go wrong with LLM-as-a-judge itself. There's a cheat sheet at the end if you want the short version.
What is LLM-as-a-judge?
Some outputs are easy to validate. A SQL query either returns the right number on the fixture or it doesn't. Many outputs can't: a summary, a support reply, an answer to "which customers are at risk of churning?". There's no single right string to compare against, and having people read thousands of outputs doesn't scale.
LLM-as-a-judge fills the gap. You give a model the input, the output and a rubric, and ask for a verdict. It comes in two flavors: pointwise, grading one output as a score or pass/fail, and pairwise, picking the better of two outputs for the same input.
The MT-Bench paper (Zheng et al., NeurIPS 2023) made it a standard tool. Its headline result explains the popularity: strong judges like GPT-4 agreed with human preferences more than 80% of the time, about as often as humans agree with each other, at a fraction of the cost and fast enough to run on every change.
GPT-4 is old now, and newer studies are more mixed. Judge's Verdict (2025) tested 54 models, 43 open and 11 closed, on scoring RAG and agent answers against ground truth. Half reached human-level agreement, and training mattered more than model size. A strong model is not necessarily a strong judge, and a smaller, cheaper model can do the job.
An example
Back to the SQL agent. A user asks, "Which customers are at risk of churning?" The agent runs a query and writes a short answer. There's no gold answer, so we write the rubric most people would write first:
You are grading the answer of a data analyst agent.
Question: {question}
Answer: {answer}
SQL that was executed: {sql}
Score each criterion from 1 to 5:
1. Definition: does the answer state how "at risk" was defined?
2. Support: is every claim in the answer backed by the SQL?
3. Usefulness: could a retention team act on this?
Reply with JSON: {"definition": n, "support": n, "usefulness": n, "reason": "..."}And the judge returns:
{
"definition": 4,
"support": 5,
"usefulness": 4,
"reason": "Defines at-risk as no order in 90 days, lists the top 20 by lifetime value, and the SQL matches."
}Run that over 500 answers and you have a dashboard. The stats look reasonable, but each drawback below breaks them in a different way. Four are weaknesses of the judge. The fifth is a mistake in how we use it.
1. Position bias
Now compare two versions of the agent on the same question, and order starts to matter. In pairwise grading, judges tend to favor one slot, often the first. The MT-Bench paper found that even GPT-4 gave a consistent verdict after the answers were swapped only about 65% of the time, and Wang and colleagues showed that reordering a pair could change which of two models won.
People show the same primacy effect (Mantonakis et al., 2009), which is why the human labels you calibrate a judge against need shuffled order too.
Each verdict looks reasonable on its own, so the bias only shows across orders. Judge every pair several times with the order shuffled, count a win only when the verdicts agree, and track the tie rate as a measure of how noisy the judge is.
2. Verbosity and style bias
Judges can favor long, confident, well-formatted answers. In our example, an agent that pads its churn answer with a paragraph of caveats and a bulleted summary can outscore one that says "customers with no order in 90 days, top 20 attached" and is right.
People have the same bias, and judges learn it from human preference data. Singhal and colleagues found that reward models trained on human preferences lean on length, and that a reward based on length alone reproduced most of RLHF's gains. Hosking and colleagues found that annotators rated more assertive answers as having fewer factual errors. When you label by hand, include a few padded-but-wrong and terse-but-right answers to see whether your own labels reward style.
Formatting may matter even more. Judging the Judges (2026) compared nine debiasing strategies and found style bias, a preference for markdown over plain prose, was the largest bias it measured, larger than position bias.
This bites hardest when you optimize against the judge: tune a prompt against a judge that likes length and you get longer answers, not better ones. AlpacaEval hit this and added a length-controlled version. Tell the judge not to reward length or formatting, and check whether your scores track token count. If they do, you're measuring verbosity.
3. Self-preference
A judge can favor outputs that look like its own, though the evidence is mixed. The MT-Bench paper listed self-enhancement as a possible bias but said its data couldn't settle it. Panickssery and colleagues found that models can recognize their own generations and that this correlates with how much they favor them. In their label-swap test on XSum, GPT-4 preferred its own summary 73% of the time, but only 32% when the labels were swapped so that it looked like another model's. Wataoka and colleagues suggest familiarity as a mechanism: text with lower perplexity under the judge scored higher, whoever wrote it.
In our example, the agent and the judge are probably the same model, because it's the one you already have a key for. Even a modest effect then rewards a style the judge happens to like. Use a judge from a different model family, or check the scores against a second judge and read where they disagree.
4. Sensitivity and drift
The judge is a prompt on a model, and both move your numbers. Reword the rubric or change the scale and the scores shift. Run the same judge twice on the same answer and you may get two different scores. Our rubric makes this worse twice over: the difference between a 4 and a 5 on "usefulness" has no definition, and the reason comes after the scores, so the judge picks a number and then justifies it.
Then there's drift. If the provider updates a hosted judge, your history shifts with no change on your side, and last month's 4.1 isn't comparable to this month's. Use pass/fail criteria with written definitions, ask for the reason before the verdict, pin the judge version, and when you must change it, rerun old outputs to measure the break. It's the model drift from the previous post, applied to the grader.
5. Using it where code should decide
This one is a misuse, not a weakness of the judge. The "support" criterion asks whether the answer matches the SQL, and code can check that by comparing the customers named in the answer with the rows the query returned. A judge is the wrong tool for checkable questions, and a better judge wouldn't change that.
A judge also fails at checking in a predictable way. It reads text, so it doesn't run the query or know the schema's traps. Show it the 320.00 net revenue answer from the previous post, where the agent counted cancelled orders, with a clean query and a confident explanation, and it will often approve it. The error is a missing WHERE o.status IN (...) clause, and an absence is harder to spot than a mistake. On JudgeBench, which pairs plausible answers with correct ones on hard questions, many strong models, including GPT-4o, did only slightly better than random.
So run the code first and give the judge only what's left. When you have a reference answer, give it to the judge and measure whether it helps: AgentJudgeBench (EMNLP 2026) found it slightly lowered agreement for two frontier judges.
The rubric, fixed
Applying the fixes to the original rubric: "support" moves to code, the scores become pass/fail with written definitions, the reason comes first, and the judge is told what not to reward.
You are grading the answer of a data analyst agent.
Question: {question}
Answer: {answer}
For each criterion, write a one-sentence reason, then pass or fail.
Do not reward length, formatting or confident tone.
1. Definition: pass only if the answer states a rule and a time window
for "at risk" (e.g. "no order in 90 days"). Fail if it only says "at risk".
2. Actionable: pass only if it names specific customers or segments
and a next step a retention team could take without follow-up questions.
Reply with JSON:
{"definition": {"reason": "...", "pass": true},
"actionable": {"reason": "...", "pass": false}}A few lines of code now check that every customer named in the answer appears in the query's result rows, and the judge, from a different model family than the agent, pinned to a version, only sees what code can't decide.
Cheaper Judges
A judge is cheaper than people, but not cheap. Every call carries the question, the answer and the rubric, and the judge writes its reasoning back. The fixes above multiply the calls: two criteria in separate calls, each run three times, on 500 answers is 3,000 calls for one eval run. Run that on every change to the agent with a reasoning model as the judge, and the eval bill can rival the product's.
Self-hosting an open-weight judge changes the math. You pay for GPUs instead of tokens, the data stays in house, and the drift from drawback 4 mostly goes away, because the weights change only when you change them. Open models are increasingly good enough: in AgentJudgeBench, the open-weight GPT-OSS-120B was the judge most aligned with people. A good benchmark score doesn't guarantee a good score on your task, so compare two or three candidates on a sample you labeled yourself.
Decision models are another option. Reward models and newer general-purpose ones like TypeSafe AI's Jev take an input and a typed question and return a label with a probability. A typed answer replaces "ask for JSON and hope", and a calibrated probability lets you route uncertain cases to a person. But a model asked "is this answer supported by the SQL?" is still judging text, so the rule from drawback 5 holds. These models are far less studied than chat-model judges, so measure them like any other.
So should you use one?
Yes, for the part code can't check, and only after you've graded the grader. Label about 100 outputs by hand, in random order, and include a few answers you know are wrong. Compare the judge's verdicts to yours with percent agreement and Cohen's kappa, since agreement alone looks good when most answers pass. If the judge passes your known-bad answers or the kappa is low, fix the judge before you trust a dashboard. Re-measure on every judge or prompt change.
A judge shifts the human work rather than removing it: instead of reading 500 answers, you read 100 and check that the judge reads them the way you would.
Cheat sheet
| Drawback | Symptom | Quick test | Fix |
|---|---|---|---|
| 1. Position | Verdict flips when the order flips | Judge each pair in shuffled orders | Count a win only when verdicts agree; report the tie rate |
| 2. Verbosity and style | Long or markdown answers win | Score the same answer plain and padded | Tell the judge not to reward length or formatting; check scores against token count |
| 3. Self-preference | Your own model's style wins | Compare with a judge from another family | Use a judge from a different family than the system it grades |
| 4. Sensitivity and drift | Scores move with no change on your side | Rerun the same outputs twice, and after any judge change | Pass/fail with written definitions, reason before verdict, pinned judge version |
| 5. Code should decide | Wrong answers pass because they read well | Seed a known-wrong answer and see if it passes | Check results, rows and calculations in code; give the judge only what is left |
| All | No idea how good the judge is | Label about 100 outputs by hand, order randomized, a few known-bad | Measure agreement and kappa; re-measure on every judge or prompt change |
References
- Bias: MT-Bench and Chatbot Arena (Zheng et al., 2023) for position and verbosity bias and agreement with humans; Large Language Models are not Fair Evaluators (Wang et al., 2023) for order effects; Order in Choice (Mantonakis et al., 2009) for the same effect in people; Length-Controlled AlpacaEval (Dubois et al., 2024); A Long Way to Go (Singhal et al., 2023) for length in human preference data; Human Feedback is not Gold Standard (Hosking et al., 2023) for assertiveness; Judging the Judges (Soumik, TMLR 2026) for style bias and mitigations.
- Self-preference: LLM Evaluators Recognize and Favor Their Own Generations (Panickssery, Bowman, Feng, 2024); Self-Preference Bias in LLM-as-a-Judge (Wataoka, Takahashi, Ri, 2024) for perplexity as a possible mechanism.
- Measuring judges: JudgeBench (Tan et al., 2024), strong judges near random on hard pairs; AgentJudgeBench (Verma et al., EMNLP 2026), judges on agent tool calls by difficulty; Judge's Verdict (Han et al., 2025), human agreement across 54 models; RewardBench for reward models.
- Prompting and fine-tuned judges: G-Eval, which established the criteria-plus-reasoning prompt most rubric judges still use; PandaLM, JudgeLM, Prometheus and Prometheus 2, and the empirical study of how far they generalize.
- Decision models: Introducing System One Models and Jev (TypeSafe AI). Vendor claims, not independently verified.