Evals: how you actually know it got better

Swap a prompt for a version that reads cleaner, that feels more careful, and ship it without checking anything else. That's an easy trap to fall into. A week later, if someone asks what actually got better, there often isn't a real answer, because nothing was measured. It just felt better, and that's what shipped.

That's the gap this post is about. Every post so far has been about building a piece of the system, prompting, retrieval, the loop, memory, guardrails. This one is about the step that tells you whether any change to any of those pieces actually helped, instead of just feeling like it did.

Why "it feels better" isn't enough

A model's output is fluent almost by default, and fluent text is persuasive regardless of whether it's actually more correct. Reading two versions side by side and picking the one that sounds better is a real bias, not a minor one, humans are reliably fooled by confident, well-formatted answers regardless of accuracy. Without a fixed way to measure the actual thing you care about, "better" quietly becomes "reads nicer," which is a different property.

Build an offline eval set first

An eval set is a fixed collection of real inputs, ideally pulled from actual usage, each with a way to check whether the output was good. That last part matters: a set of inputs without any way to score the outputs isn't an eval set, it's just a list of examples you'll still be reading with your gut.

The scoring method depends on the task. Some things have a clear right answer, a factual question, a specific format, and those can be checked with exact-match or a simple rule. Most interesting LLM outputs don't have one clean answer, and that's where the next two methods come in.

LLM-as-judge, and where it breaks

LLM-as-judge means using a second model to score or compare outputs, cheaper and faster than a human for every single case. It's not free of problems. Judge models have known biases, they tend to favor longer answers regardless of quality, and they can favor answers similar in style to their own default voice over answers that are actually more correct but phrased differently.

The fix isn't to abandon it, it's to calibrate it: periodically check the judge's scores against real human judgment on a sample, and if they diverge, that's a signal to adjust the judge's prompt or fall back to human eval for that specific case, not a reason to trust the judge blindly forever.

Human eval: still the ground truth, still the expensive one

For anything where getting it wrong actually matters, medical, legal, financial, anything close to those, a human should be checking outputs, at least on a sample, at least periodically. It's slower and it doesn't scale to every single output the way LLM-as-judge does. That's exactly why it's usually reserved for a sample rather than everything, and used to calibrate the cheaper automated methods rather than replace them entirely.

Regression testing: catching the thing that used to work

An eval set isn't just for comparing a new version against an old one. Run it every time anything changes, a new prompt, a new model version, a new chunking strategy, and it becomes a regression test: something that used to score well and now doesn't is a signal something broke, even if nothing threw an error. Silent regressions are the most common failure in AI systems specifically because nothing crashes. The output is still fluent. It's just wrong now, or wrong in a new way, and only a fixed eval set run consistently catches that.

What this actually buys you

Once you have a real eval set and a consistent way to score it, "did this change help" stops being a feeling you argue about and becomes a number you check. That doesn't remove judgment from the process, someone still has to decide what to measure and what counts as good, it just moves the judgment call to before you ship, where it's cheap to be wrong, instead of after, where it isn't.

Evals tell you whether a change helped. They don't tell you whether the system, running right now, in production, is actually healthy. That's the last piece.