Evals That Actually Work

An eval is just a function: turn the model's output into a number you can average and trend.

Keep goldens small and mean#

type Case = { input: string; expected: string; rubric?: (out: string) => Score };

Ten sharp cases beat a thousand blurry ones. If a case never flips a run red or green, it's dead weight.

Run them on every commit#

Fast, deterministic, and checked into the repo. Flaky evals train you to ignore failures.

Weiterführend in der AI Engineering for Builders-Serie: dieser Post ist der tl;dr — die tiefe Version mit Eval-Pipeline, AI-as-Judge und Modell-Auswahl steht in Part 3 — Evaluation First. Starte am Anfang mit Part 1 — The AI Engineering Mental Model.