Evaluations are the product
If you cannot say what good looks like, you cannot ship it. A sketch of how evaluation shapes everything downstream.
It is hard to ship something when you cannot say what good looks like. With AI systems that question arrives early and keeps coming back.
An evaluation is simply a way of writing down what you expect. Once it exists, it quietly shapes almost every decision that follows.
Deciding what “better” means
Before a model is trained or a prompt is tuned, someone has to decide what counts as a better answer. That decision is the real specification, whether or not it is ever written down.
“If you cannot measure what you want, you end up building what you can measure.”
Start with a handful of examples
Good evaluations tend to start small: a handful of real examples, read carefully, with notes on what went right and wrong. Over time they grow into something that can be run often and compared across versions.
Part of the product, not the last step
Treating evaluation as part of the product, rather than a step at the end, changes what gets built. It keeps the work honest about what it is actually trying to do.