Your eval is a measuring instrument

Someone changes a prompt. The score goes from 71 to 74. Is the product better?

Nobody knows. So three people try it on the examples they happen to remember, agree it feels better, and ship.

That meeting is the failure. Not the model. The eval was supposed to settle the question, didn’t, and everything downstream got decided by vibes with a number stapled on for cover.

An eval is a measuring instrument. Uncalibrated, it produces decoration.

Start where you already know the answer

Not your most important workflow. The one you know cold — where you can look at an output and know in two seconds that it’s wrong.

That instinct is your ground truth. It’s sitting in your head where nobody else can use it. Getting it out is the whole job.

Figure 1 — Every eval starts as a judgment you already make in your head.

Pick a domain you don’t know and you’re building the instrument and learning the subject at the same time. When they disagree, you can’t tell which one is broken.

Study the traces, not the questions

Users don’t arrive one turn at a time. They ask something vague, get half an answer, refine it, come back.

So write down what good looks like at each step — and separately, what good looks like for the whole run.

Figure 2 — Users don’t arrive one turn at a time, so don’t grade one turn at a time.

You need both. Step checks tell you where a run broke, which is what you need to fix it. The outcome tells you whether it mattered, which is what you need to prioritise. Grade only the steps and you optimise for a tidy transcript. Grade only the outcome and you know you’re broken without knowing where.

Then break them on purpose

Traces collected from a working system are traces where things went right. That’s not where products fail.

Figure 3 — The happy path is the easy part. Write the traces where things arrive broken.

Empty the tool result but keep the success status. Truncate the retrieved chunk mid-sentence. Strip the disambiguating noun out of the request.

For each one, write the failure you expect and the behaviour that counts as passing. “Said it found nothing” is a pass. “Produced something fluent” is not — and if your rubric can’t tell them apart, it will score the second one just as highly.

Write the rubric before you pick the instrument

The first question people ask is usually “should we use an LLM judge?” That’s the second question.

First: what does good look like, written down tightly enough that two people grade the same output the same way.

Figure 4 — Write the rubric first. The instrument you pick will otherwise rewrite it for you.

Order matters, because instruments are opinionated about what they can see. Pick the judge first and the rubric drifts toward what a judge scores consistently. Pick string matching first and it drifts toward exact-answer questions. Either way the criteria you actually cared about fall off the list, and nothing in the dashboard records that they’re gone.

More of it is script-checkable than you’d expect. Did the citation resolve. Was the tool called with the arguments the case specified. Those are assertions, not judgments — free, exact, and they don’t drift.

Quality first. Cost next.

Treat evals like frontier models: establish the quality frontier, then work your way down the cost curve.

Version one should be the most expensive thing you can stand to run. Pay the annotators. Use the big judge model. Read fifty transcripts yourself.

Figure 5 — Establish the frontier first. Walking down the cost curve is the second problem.

Cost reduction needs a reference. Swap in a smaller judge and the only way to know what you lost is to compare against a measurement you already trusted.

Start cheap and nothing ever tells you the instrument is broken. The dashboard reads 71, then 74, and nobody has grounds to doubt either number.

Then walk left. Distil the judge and measure agreement against the expensive one. Sample on commits, run the full set on releases. Convert every judged criterion that a script could decide.

Name the failure

This is the first thing to build once you have v1. Pull the last 500 or 1,000 interactions, find the failures, cluster them, name the clusters.

Get specific. “Bad answer” isn’t a cluster name, it’s a shrug.

Figure 6 — Sixty-three failures. One pile teaches you nothing; five named clusters are five work items.

Wrong document retrieved. Right document, wrong section. The context was there and it answered from memory anyway. Should have punted, made something up. Question was ambiguous and it guessed instead of asking.

Five bugs, three owners, five different fixes. Average them into one quality score and you’ll spend a quarter improving retrieval while the damage is coming from ungrounded confidence.

A named failure also converts straight into a test. Once “right document, wrong section” exists as a category, you can write ten cases where the answer lives in §4.2 of a document whose §1 looks similar, and assert on the retrieved section ID rather than on the text of the answer. Cheap, deterministic, fails for exactly one reason. You can’t write it until you’ve named it.

Four kinds, four jobs

Figure 7 — Four kinds of eval, doing four different jobs. You need all four.

Hill-climb — the frontier of what the product can do. Short shelf life by design: the week you pass them all, they stop telling you anything. Refresh constantly.

Regression — did we break today’s product while climbing? Boring, almost always passes, should block. The urge to prune them is the urge to cut the net down because nothing has fallen into it.

Smoke — not hard, just can’t be wrong. Product identity. The obvious refusal. Does it come up at all. Every single run.

Launch — closest to real traffic, least control, most realistic. The last thing that can still surprise you, because everything upstream is made of situations you already thought of.

Keep it running, and keep it current

Two things kill a suite.

Friction. If a full run needs an afternoon of someone’s attention, it runs when someone has an afternoon, which is never the moment it would have mattered. One command, fixed seed, fixed set, triggered by merges rather than by good intentions.

Drift. The cases you wrote in March measure March. The suite keeps returning confident numbers the whole time, which is exactly what makes it dangerous.

Figure 8 — An eval that needs a volunteer runs roughly never — and the one you wrote in March is already stale.

Sample last week’s traffic, cluster the new failures, add cases, retire the dead ones. Monthly.

Watch the mix while you do it. If 90% of your set came out of the failure taxonomy, your headline number describes edge cases rather than users.

What I don’t have a good answer for

The judge ceiling. Strong judges agree with human preference around 80% of the time — roughly the rate at which humans agree with each other. Your judge is about as good as a second annotator, and that floor isn’t moving.

Self-preference. Models identify their own outputs at better than chance, and the strength of that recognition tracks how much they favour those outputs. Randomising presentation order doesn’t touch it.

Multi-turn credit assignment. A small unstated assumption at turn one that becomes a confidently wrong answer at turn four passes every step check you thought to write.

Labels. Every technique here consumes human judgment somewhere. Distilling the judge moves that cost around rather than removing it.

Goodhart. The moment the score is a target, it starts capturing effort that would otherwise have gone into the product. Keep a holdout nobody optimises against, and be suspicious of any number that only moves up.