An eval harness that still means something after you change the prompt

If a prompt tweak can move the score more than a model swap, the leaderboard is measuring your wording, not the system.

We had a neat spreadsheet and a rising curve, and then someone shortened the system prompt. The score jumped four points. Nobody had improved the model.

A harness that survives a prompt change needs three boring properties. First, the task set is versioned and frozen; you can add a new set, you cannot silently edit the old one. Second, the scorer is not another large model judging in free text, or if it is, you also keep a deterministic check for the subset that has a right answer. Third, every run records the exact prompt template, decoding settings, and tool policy next to the score.

I now keep two numbers on the same plot: a closed, automatically scored set, and a smaller human-reviewed set. If they disagree, I do not average them. I go read the diffs.

The goal is not a perfect number. The goal is that a week later you can tell whether the system got better, or you just asked the question more nicely.