Green Eval, Wrong Answer: Eleven Ways an Agent Eval Lies

A SQL agent is asked the simplest question finance can ask: what was our net revenue last month? It answers 320.00. The right answer is 210.00. Nothing errors, the query looks plausible, and the eval score is green.

Everyone talks about evals now. Model launches lead with benchmark tables, agent products call themselves eval-driven, and there's a growing shelf of eval platforms. It sounds like a solved problem: write some test cases, get a score, ship. It isn't. An eval score can look healthy and correct while the agent is wrong, and there are many ways for that to happen.

The failure modes come in two kinds. Some are inherited. Classic machine learning and software testing met them long ago and named them: proxy metrics, overfitting the validation set, train-serve skew, concept drift, the flaky test. If you've shipped a model before, you already know the defenses. Others are new, because an agent is a different beast. It doesn't give the same answer twice. It takes many steps, with costs and side effects. It's often graded by another model with biases of its own. And sometimes the right output is a question back to the user.

The running example is that SQL agent, which answers business questions about an e-commerce database, and that net revenue question. We'll build its eval one step at a time, and each step adds a new way to be fooled.

Note: this is a constructed example. The scores are invented to make the mechanics visible. The failure modes are real.

What are we measuring?

An eval exists to support decisions: is this good enough to ship, did the last change help, which of two options wins, and where does it fail. Without one, those calls rest on a handful of examples someone tried by hand. Correctness is the obvious thing to measure, and the one most evals stop at. For an agent it's rarely enough:

• Correctness: is the answer right? For a SQL agent, is the number right, not just the query plausible?
• Cost and time: tokens spent by the model, compute spent by the database, and how long the user waits. An agent that answers correctly after scanning a terabyte isn't free, and latency grows with every retry and tool call. (Step 4)
• Reliability: does it give the right answer every time, or only most times? (Step 3)
• Safety: does it stay inside its permissions? No writes, no unbounded scans, no data the user shouldn't see. (Step 4)
• Behavior: when the question is ambiguous, does it ask or state its assumption instead of guessing silently? (Step 6)

These trade off against each other. A bigger model may be more correct and slower. More retries may be more reliable and more expensive. So decide upfront which are gates and which are budgets. For a finance question, correctness and safety are gates: fail either and nothing else matters. Cost and time are budgets: they should stay under a limit, and within it, lower is better. Blending everything into one score hides exactly the trade-offs you need to see. Most of the failure modes below are about correctness, because that's where evals lie most convincingly, but cost, reliability and behavior come back along the way.

The system under test

The agent gets a natural-language question and writes SQL against five tables: customers, orders, order_items, refunds and products. The schema is deliberately boring, apart from the highlighted fields, which are the traps.

Orders have a status, and not every status is revenue: cancelled and pending orders exist. Timestamps are stored in UTC while the business reports in New York time. Discounts live on the line item. Refunds are partial and attach to the order, not to an item. None of this is exotic. It's what a real schema looks like after a few years.

The question, and two answers

Here is a reference query for "what was our net revenue last month?". It keeps only paid, shipped and delivered orders, computes the month boundaries in New York time, subtracts discounts per line, and nets out refunds after aggregating them per order.

WITH bounds AS (
  SELECT
    (date_trunc('month', now() AT TIME ZONE 'America/New_York') - interval '1 month')
      AT TIME ZONE 'America/New_York' AS lo,
    date_trunc('month', now() AT TIME ZONE 'America/New_York')
      AT TIME ZONE 'America/New_York' AS hi
),
month_orders AS (
  SELECT o.id,
         SUM(oi.quantity * oi.unit_price_cents - oi.discount_cents) AS gross_cents
  FROM orders o
  JOIN order_items oi ON oi.order_id = o.id
  CROSS JOIN bounds b
  WHERE o.status IN ('paid', 'shipped', 'delivered')
    AND o.created_at >= b.lo AND o.created_at < b.hi
  GROUP BY o.id
),
refunded AS (
  SELECT order_id, SUM(amount_cents) AS refund_cents
  FROM refunds
  GROUP BY order_id
)
SELECT SUM(m.gross_cents - COALESCE(r.refund_cents, 0)) / 100.0 AS net_revenue
FROM month_orders m
LEFT JOIN refunded r ON r.order_id = m.id;

And here is what an agent plausibly writes:

SELECT SUM(oi.unit_price_cents * oi.quantity) / 100.0
FROM order_items oi
JOIN orders o ON o.id = oi.order_id
WHERE o.created_at >= date_trunc('month', now()) - interval '1 month'
  AND o.created_at <  date_trunc('month', now());

It ignores status, discounts, refunds and timezone, and it returns a clean, plausible number. Take a small trap fixture: three September orders, one of them cancelled and one with a discount and a partial refund.

Order	Status	Gross	Discount	Refund	Reference counts	Agent counts
A	delivered	100	0	0	100	100
B	shipped	150	10	30	110	150
C	cancelled	70	0	0	0	70
Total					210	320

The reference returns 210.00 and the agent's query returns 320.00. Nothing errors. A dashboard built on it would look fine for months.

Step 1: Match the SQL text

Start simple. Write 50 questions with reference SQL and check whether the agent's query matches. The score is 38%. Read the failures and most of them aren't failures: a subquery where the reference used a CTE, a column aliased differently. A string comparison can't tell those apart from a wrong answer.

This is measuring a proxy. Many different queries are valid for one question, so surface similarity punishes style and says nothing about correctness.¹

Inherited from classic ML: judging machine translation with BLEU, or summaries with ROUGE. Same disease. You reward overlap with one reference when many outputs are right.

Step 2: Run on one thin database

The obvious fix is to run both queries and compare the results. This is execution accuracy, and the score jumps to 84%. Everyone relaxes. But the thin test database has 200 rows, no refunds in the test month and no cancelled orders, so the agent's query that ignores both returns exactly the right number. It passes for the wrong reason.

Exact match gave false negatives. Execution accuracy on a thin database gives false positives, and those are more dangerous because they look like progress. The fix is to stop relying on one database and distill a suite of databases that, between them, separate the reference from plausible wrong answers.¹ For us that means trap fixtures seeded on purpose: cancelled orders, partial refunds, multi-item orders, an order placed at 23:30 New York time on the last day of the month, a discount that zeroes out a line.

Here is a second question the thin database misses: refund rate by product category. The agent's version looks reasonable.

SELECT p.category,
       SUM(r.amount_cents)::numeric
         / SUM(oi.quantity * oi.unit_price_cents - oi.discount_cents) AS refund_rate
FROM order_items oi
JOIN orders o   ON o.id = oi.order_id
JOIN products p ON p.id = oi.product_id
LEFT JOIN refunds r ON r.order_id = o.id
WHERE o.status IN ('paid', 'shipped', 'delivered')
GROUP BY p.category;

Refunds attach to the order, so this join repeats the order's refund on every line item. On the thin database, where every order has one item, nothing is off. On the trap fixture, which has a multi-item order, the same refund is counted once per line and charged to every category in the order. The query reports about 31% for books and 50% for toys, when the right figures are about 16% and 25%. The reference aggregates refunds per order first, then allocates them to items in proportion to line revenue.

Inherited from classic ML: Clever Hans, the horse that seemed to do arithmetic but was reading his trainer's body language. A model can score well for reasons that have nothing to do with the task. If the test data never exercises the hard cases, a passing score tells you nothing about them.

Step 3: Run each question once

The suite is green and the agent gets the net revenue question right. Run it again and it might not. A classic classifier is a function: same input, same output. An agent samples. Ask the same question ten times and you can get ten different queries, say seven that filter on status and three that don't. Temperature 0 doesn't fully save you, because serving-side effects like batching mean identical requests can still produce different outputs. And an agent compounds it: one different token early changes which tables it inspects, and every step after that.

So one run per question is a sample of one. A 90% score from a single run can't tell an agent that's always right on 90% of questions from one that's right on every question 90% of the time, and for a finance dashboard those are very different products. Run each question several times and report the pass rate per question. For questions that must work every time, require all k runs to pass. τ-bench calls this pass^k. It found that even state-of-the-art function-calling agents such as GPT-4o succeeded on fewer than half the tasks, with pass^8 below 25% in its retail domain.

The same applies when comparing two versions of the agent: a two-point gain on 50 questions, one run each, is well within the noise.

New with agents: classic models are deterministic at inference time. Random seeds made training noisy. Agents make every prediction noisy, and that variance is part of what you're measuring, not noise to average away.

Step 4: Grade only the final answer

An agent doesn't write one query. It lists tables, samples rows, tries a query, reads the error, and tries again. The eval checks the number at the end. Now open a trace that passed. The agent ran fourteen queries, scanned all of order_items three times with no filter, and at one point tried to CREATE TABLE tmp_revenue to stage its results. The answer was correct. On a warehouse billed by bytes scanned, that's a bill. On a production replica with write permissions, it's an incident.

The final answer can also disagree with the work. Agents summarize what they did, and the summary can describe a filter the query never applied: "excluding cancelled orders" on top of SQL that doesn't mention status. If you grade the prose, you grade the claim, not what actually ran.

Cost needs the same attention, even when the final query is right. Style changes the text of a query. Inefficiency changes its cost, and you can't tell them apart by reading the SQL, so you measure. Run the agent's query and the reference on a scaled copy of the data and compare the query plan, the rows or bytes scanned, and the runtime. If the plans match, the difference is style: a CTE and a subquery usually compile to the same plan, since Postgres 12 inlines a CTE that is referenced once. If the cost differs by a large factor, it's inefficiency, even though the results are identical.

Scale matters here. On 200 rows a correlated subquery and a join both finish in a couple of milliseconds. At 80 million order_items rows one of them can take tens of seconds and the other about one. A thin dataset hides inefficiency as well as logic errors.

Three details make this workable:

• Use a ratio, not an absolute number. Compare the agent's cost to the reference's, so the eval survives hardware changes.
• Let the agent beat the reference. The reference can be inefficient too.
• Ban a few patterns outright. Some are wrong whatever the plan says: a function wrapped around an indexed column in a WHERE clause, an accidental cross join, SELECT * on a columnar engine billed by bytes scanned.

BIRD, a large text-to-SQL benchmark, does something similar with its Valid Efficiency Score: wrong results score zero, and correct ones are scored on runtime relative to the reference.

So grade the trajectory as well as the outcome. Give each task a budget for steps and bytes scanned. Run the eval with the same read-only credentials the agent gets in production, in a sandbox that looks like production. Keep a list of actions that fail a task whatever the answer: DDL, DML, unbounded scans of large tables. And check the stated answer against the SQL that executed.

New with agents: a classifier's output is its prediction. An agent's output is a sequence of actions, and those actions have costs and side effects that the final answer doesn't show.

Step 5: Trust the LLM judge

Real questions don't all come with a gold query. "Which customers are at risk of churning?" has no single right answer, so you add an LLM judge. Now the judge is the problem. Judges tend to like long, polished, heavily commented queries and rate the terse correct one as merely fine. They tend to prefer whichever answer comes first, and may favor answers in their own style. The MT-Bench paper that popularised LLM-as-a-judge documented position and verbosity bias, flagged self-enhancement as a possible third, and found that even GPT-4 gave the same verdict after the answers were swapped only about 65% of the time.

Position bias isn't unique to models. In a wine-tasting study, participants given a sequence of samples tended to pick the first as their favorite, even though every sample was the same wine. That's why careful human evaluations randomize the order, and why your human labels need it too when you use them to calibrate the judge.

None of this makes judges useless. The same paper found GPT-4's agreement with human preferences above 80%, roughly the level at which humans agree with each other. It means a judge is an instrument that needs calibrating. Swap the order and require both verdicts to agree. Give it a rubric. Have it grade the result set and the query plan instead of the prose of the SQL. Measure its agreement with human labels on a sample of a hundred or so, and measure again when you change the judge model.

New with agents: the grader is now the same kind of system as the thing it grades, with the same tendencies, possibly including a taste for its own style. The fix is borrowed from inter-annotator agreement. You wouldn't trust a labeler you hadn't measured. Don't trust a judge you haven't either.

Step 6: Never test the ambiguous questions

Some questions can't be answered correctly as asked. "How many active customers did we have last quarter?" uses a term that appears nowhere in the schema. The agent picks "ordered at least once". Finance means "ordered in the last 90 days and not fully refunded". No amount of SQL skill fixes that, and comparing against a gold query won't catch it, because the gold query just encodes one of the two readings.

What you can evaluate is behavior. Put ambiguous questions in the set on purpose, and score whether the agent asks a clarifying question or states its assumption in the answer. A silent guess fails, even when it happens to match.

A related remedy is a confidence signal: have the agent report how sure it is that the question maps cleanly onto the schema and the known definitions, and route low-confidence questions to a clarifying question or a human. Self-reported confidence is often poorly calibrated, so evaluate the signal itself: when the agent says it's confident, is it right? A score that is high on the wrong answers is worse than none.

Inherited: classic ML met this as out-of-vocabulary words and out-of-distribution inputs, such as extreme values the training set never contained. The classic remedies were confidence thresholds and a reject option: abstain when unsure. What's new is that an agent can do better than abstain. It can ask.

Step 7: Report one number

The dashboard says 91%. Cut by question type, the picture changes.

Question type	Share of eval set	Accuracy	Share of real traffic
Simple aggregations	62%	98%	25%
Joins and filters	28%	90%	30%
Date and timezone logic	7%	55%	25%
Refunds and net revenue	3%	40%	20%

Weighted by the eval set, that's 91%. Weighted by the questions people actually ask, the same agent scores about 73%. The eval set over-represents easy aggregations and under-represents date logic and refunds, which are the questions where a wrong answer costs the most. The numbers are invented, but the shape is common.

The mismatch isn't always a mistake. You may oversample hard questions on purpose, because that's where the agent fails and where you need enough cases to measure anything. That's fine, as long as the headline is adjusted. Classic ML does the same when it trains on a balanced sample and then corrects for the real base rates. So report per slice, reweight the headline by the traffic you expect, and watch the worst slice instead of the mean. The traffic weights only exist if you log production questions, which comes back in step 10.

Inherited from classic ML: inside the eval set, it's class imbalance, the reason we report per-class metrics. Between the eval set and production, it's a test set that doesn't match the population it's meant to represent. The remedy is old too: weight each slice by its share of real traffic, the same correction survey statisticians call post-stratification.

Step 8: Tune against the test set

Now the iteration begins. Date logic scores low, so you add schema hints and few-shot examples to the prompt, and the score climbs. Look closely and half the examples are paraphrases of eval questions. The score now measures how well the prompt memorised the eval set. That's Goodhart's law: when a measure becomes a target, it stops being a good measure.

The remedy is the old three-way split. An agent has no training run, but the roles still map:

• Training set: everything that goes into the agent. Few-shot examples, schema hints, and the questions you debug against by hand. Anything that appears in a prompt lives here.
• Validation set: the questions you score every change against while iterating. You will overfit it a little. That's expected, and it's why it can't also be the number you ship on.
• Test set: held out, refreshed from real traffic, and run rarely: before a release or a model switch. Nobody tunes against its failures. If a test question gets copied into a prompt, it moves to training and gets replaced.

Without the split, the eval that drives iteration is also the one that decides shipping, and it ends up grading its own homework.

Public benchmarks leak in a way your own test set doesn't. Spider and BIRD have been online for years, so their questions and gold queries may be in the model's training data, and a strong public score can be memory. That's one more reason a held-out set built from your own traffic matters more than a leaderboard.

Inherited from classic ML: overfitting the validation set, and the reason the test set is touched only at the end. What's new is that the model may have seen the public test set before you did.

Step 9: Trust the labels

Gold queries are written by people, and people make mistakes. Northcutt and colleagues found label errors in the test sets of ten widely used machine learning benchmarks, ImageNet among them, enough, they argued, to destabilize benchmark rankings. Text-to-SQL is no better. A January 2026 study reported annotation error rates of 52.8% in BIRD's Mini-Dev split and 62.8% in Spider 2.0-Snow, and showed that correcting a subset of BIRD's dev set moved agents' leaderboard positions by up to nine places in either direction.

Some label problems aren't mistakes at all, they're definitions. Whether a refund counts against the month of the order or the month of the refund is a business decision, and the net revenue reference above quietly picks the first. The refund-rate reference makes another choice when it allocates refunds by line revenue. If your gold queries embed definitions like these, the metric is pinned in the wrong place. Ideally the definition lives in a semantic layer that both the agent and the eval read, so "which definition is right" has one owner.

Your 50 hand-written gold queries deserve the same suspicion as a public benchmark. Have a second person verify them, and when a strong agent and the reference disagree, treat it as a bug report on the reference first.

Inherited from classic ML: noisy labels. You'd audit a labeling vendor before trusting its data. Audit your own gold queries the same way.

Step 10: Ship it, and let it drift

Production doesn't look like the eval. Users ask vague follow-ups like "and excluding the promo?", which only make sense with the previous turn. They use words the schema doesn't contain. They paste in column names from a dashboard instead of the database. The eval set was written by someone who knows the schema, one clean single-turn question at a time. This is train-serve skew, and it's one of the most common reasons an eval number doesn't transfer. The remedy is dull and effective: log real questions, sample them, and make them the eval set.

And even a good eval is a photo of a moving target. Five things drift, and each needs its own defense:

What drifts	Example	Defense
Schema	A payments table is split out or a column is renamed, and yesterday's few-shot examples reference columns that don't exist	Diff the schema on every migration and rerun the suite
Data	A partially_refunded status or a second currency appears, and correct queries become silently wrong	Daily canary queries with known answers; track error rate, empty-result rate, result size and cost per query
Definitions	"Active customer" goes from 30 days to 90	Version definitions in the semantic layer and rerun the suite when they change
Questions	A new team starts asking things nobody put in the eval set	Keep sampling production questions into the eval set
Model	The provider updates the model, or you swap the judge, and scores move with no change on your side	Pin model versions where you can, and rerun the suite when they change

Classic ML has names for most of these: covariate shift when the inputs change, concept drift when the right answer to the same input changes. Concept drift bites hardest: Google Flu Trends estimated flu activity from search queries, and by early 2013 it was predicting more than double the proportion of doctor visits for influenza-like illness that the CDC reported (The Parable of Google Flu). A measure that was validated once is not validated forever.

Above all, keep that loop running, so every production failure becomes a test case.

Inherited: train-serve skew and drift are old. Two things are new. In a multi-turn agent the input includes everything said before, so the eval needs multi-turn cases, not just single questions. And with a hosted model, an update you didn't make can move your scores, and your judge, at the same time. In classic ML you retrained on your own schedule.

Step 11: Nobody reads the failures

The score dips one week. Someone lowers the threshold from 90 to 85 so the pipeline goes green again. Nobody opens the 40 failed traces, which would have shown that 31 of them involve refunds. An eval that doesn't gate anything, or whose gate moves whenever it's inconvenient, is a dashboard.

Error analysis, meaning actually reading the failures and sorting them into categories, is the cheapest and highest-value habit in this whole post. A score tells you something changed. Reading the traces tells you what.

Inherited: the flaky test that gets rerun until it passes, or skipped. As Will Keleher puts it, that test isn't flaky, it's broken. An eval case that fails only some of the time is the same signal, and for an agent it's literally what Step 3 measures: a pass^k below one.

A checklist for a trustworthy eval

Before you trust the score

• ☐ Every metric the eval checks has one written definition, with an owner, that both the agent and the eval read. (Step 9)
• ☐ Answers are checked by executing queries on fixtures seeded with known traps, never by matching SQL text. (Steps 1–2)
• ☐ Every question runs at least five times, and must-pass questions pass all of them. (Step 3)
• ☐ Each task has a budget for steps and cost, runs under the production read-only credentials, and fails on any write. (Step 4)
• ☐ The LLM judge has been checked against at least 100 human labels, with answer order randomized for both, and is re-checked whenever the judge model changes. (Step 5)
• ☐ The set contains ambiguous questions on purpose, and a silent guess fails them. (Step 6)
• ☐ Scores are reported per slice, the headline is reweighted by production traffic, and the worst slice is shown next to it. (Step 7)
• ☐ A second person has verified every gold query. (Step 9)

Before you ship

• ☐ Prompt examples, the validation set and the test set don't overlap, and the test set hasn't been tuned against. (Step 8)
• ☐ Model and judge versions are pinned, and the suite has been rerun on the exact versions you're shipping. (Step 10)
• ☐ The pass threshold is written down, and lowering it requires a written reason. (Step 11)

After you ship

• ☐ A sample of production questions is added to the eval set every month. (Step 10)
• ☐ Every schema migration triggers a rerun, and canary queries with known answers run daily. (Step 10)
• ☐ Every failed case has a named owner who tags it with a cause within a week, and the tag counts are reviewed with the score. (Step 11)
• ☐ A case that fails only sometimes gets investigated, not rerun until it passes. (Steps 3 and 11)

Failure modes, side by side

Step	Failure mode	Kind	Classic ancestor, or what's new
1	Matching the reference SQL text	Inherited	BLEU and ROUGE
2	One thin test database	Inherited	Clever Hans
3	One run per question	New	Outputs are sampled, so measure pass^k
4	Grading only the final answer	New	Many steps, with costs and side effects
5	An uncalibrated LLM judge	New	The grader is a model; calibrate it like an annotator
6	Not testing ambiguous questions	Inherited	Out-of-vocabulary and out-of-distribution inputs; the agent can ask
7	One headline number	Inherited	Class imbalance; a test set that doesn't match production
8	Tuning against the test set	Inherited	Train, validation and test leakage; Goodhart's law; public-benchmark contamination
9	Trusting the labels	Inherited	Noisy labels
10	Production skew and drift	Inherited	Train-serve skew, covariate shift, concept drift; plus multi-turn cases and model updates
11	Nobody reads the failures	Inherited	The skipped flaky test

Know them, plan for them

Eight of the eleven failure modes are old, which is reassuring. If you've run ML in production, you already know the defenses, and they still work. The three new ones are where the work is. It's tempting to treat an agent like a classifier that outputs text: one input, one run, one grade on the final answer. Most of the new failure modes live in the gap between that picture and what an agent actually does.

You don't need all eleven defenses on day one. You do need to know the failure modes, so you recognize a score that looks too good. Plan first for the ones your agent is most exposed to: a finance agent needs trap fixtures and pinned definitions, while an agent that talks to customers may need ambiguity handling and a calibrated judge. Monitor for the rest once you ship, because skew and drift only show up in production. Adjust the numbers to what a wrong answer costs you.

An eval isn't a score you earn once. It's an instrument, and like any instrument, it's only as good as its last calibration.

---

Notes

1. The text-to-SQL community has measured both problems. Work on test-suite evaluation found that Spider's official exact set match metric had a false negative rate of about 2.5% on average, and up to 8.1% in the worst case. The same paper proposed distilling a suite of databases as the fix for single-database false positives. ↩

Sources

• Semantic Evaluation for Text-to-SQL with Distilled Test Suites (Zhong, Yu, Klein, EMNLP 2020): false negatives of exact match, false positives of single-database execution accuracy.
• Can LLM Already Serve as A Database Interface? (BIRD): execution accuracy and the Valid Efficiency Score.
• τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (Yao, Shinn, Razavi, Narasimhan, 2024): pass^k and agent inconsistency across runs.
• Defeating Nondeterminism in LLM Inference (Thinking Machines Lab, 2025): why temperature 0 still isn't deterministic.
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena: position and verbosity bias, possible self-enhancement bias, and agreement with humans.
• Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (Northcutt, Athalye, Mueller).
• Pervasive Annotation Errors Break Text-to-SQL Benchmarks and Leaderboards (Jin, Choi, Zhu, Kang, January 2026).
• The Parable of Google Flu: Traps in Big Data Analysis (Lazer, Kennedy, King, Vespignani, Science, 2014).