Notes on serving models, evaluating them, and keeping the machines honest. No product pitches, no launch-week essays.
Most of the cost sits in the KV cache and in bad batching, not in the parameter count you quote in a blog post.
If a prompt tweak can move the score more than a model swap, the leaderboard is measuring your wording, not the system.
Embeddings help when the query and the document do not share words. They do not excuse a messy corpus.
The feature list is often decided by what still fits after the cache, the image, and the safety window.
A request id, a model name, a token count, and a reason for the retry will save you more time than another dashboard.
Treat a new checkpoint like a schema change: small, reversible, and compared on the same traffic.