CUDA memory is a product constraint, not a hardware footnote

The feature list is often decided by what still fits after the cache, the image, and the safety window.

Product meetings like to start with features. On a generation box, I start with a memory map. Weights, activations, KV cache, vision encoder, and a reserve for fragmentation and a bad batch. Whatever is left is the feature budget.

This sounds pessimistic. It is actually freeing. Once the map is on a whiteboard, arguments about “just add a 32k context” become arithmetic. You can show what has to be dropped, or which requests have to be limited, or why a second pass over the image is not free.

One habit I recommend: log the high-water mark of allocated memory per request type, not just average utilization. Average utilization hides the one path that OOMs on Friday afternoon, when a user pastes a repo and attaches two screenshots.

Hardware is part of the product. If you treat memory as an implementation detail, the product will discover it for you, usually in front of a customer.