evals turn a demo into a product

Most LLM apps ship with zero evals — quality is judged by spot-checking, which is vibes. That bit me twice: a failure rate I claimed and then couldn’t reproduce, and a prompt typo that shipped because nothing tested the prompt’s behaviour.

So I built an eval harness: a golden set where each case is a real past failure, run through the actual shipping prompt and graded by a stronger model against a rubric. The trick is you don’t trust a single score — output is non-deterministic — you trust the delta between runs. Change a prompt, run before and after, watch the failures move. It stays out of CI on purpose (it costs real calls and is noisy); I run it by hand before any model or prompt change.

Then I automated the part I’d forget: an agent that reads real usage, judges it, and proposes new test cases for the failures — I approve them. Manual harness → an agent that grows it. That’s the line between “I called an API” and “I engineer an LLM system”: quality becomes a measured thing, not a hope.