Shipping a model change without a mystery regression

Treat a new checkpoint like a schema change: small, reversible, and compared on the same traffic.

A new checkpoint is not a new skin. It is a behavior change that will show up in support tickets two days later, usually in a corner case nobody demoed.

The release process that has hurt us least is also the least glamorous. Shadow a slice of real traffic. Score the old and new models on the same frozen eval set. Keep a switch that can go back in one flag, not a rebuild. And write down what you are willing to make worse in exchange for what you think will get better.

If you cannot name the thing that might get worse, you are not ready to ship. You are hoping.