Teams ship LLM features on vibes for longer than they admit. It works until the day a prompt change quietly breaks a case that used to be fine.
Fifty examples are enough to start
Take real requests from your logs or from the people who will use the feature. Write down the expected outcome for each. That is your first evaluation set, and it will find more problems in a day than a week of manual testing.
Every reported bug then becomes a new row. The set grows in exactly the direction your product hurts.
Score the thing you actually care about
For extraction, compare fields. For classification, use accuracy per class so a rare label cannot hide behind a common one. For open text, define two or three specific criteria such as correctness against the source and following the required format, then score those rather than asking for a general rating.
Model graders are useful and biased
Using a model as a judge scales well and drifts. Calibrate it against a set a human graded, check agreement periodically, and keep a sample of runs that a person still reviews by hand.
Run it on every change
Prompt edits, model upgrades, retrieval tweaks and library updates all change behaviour. Wire the evaluation into your pipeline so a regression shows up in a pull request instead of in a support ticket.
Keep watching after release
Track how often users accept an answer, edit it or abandon it. Those numbers tell you where reality differs from your test set, and they are what turns the next release from a guess into a decision.
