Serving a 7B model on a single GPU without pretending it is free

Most of the cost sits in the KV cache and in bad batching, not in the parameter count you quote in a blog post.

A 7B model looks modest until you try to keep a few dozen concurrent chats alive. The weights fit. The KV cache does not, at least not if every session is allowed to grow to a long context and nobody is in charge of preemption.

I have been running a small internal assistant on one 24GB card. The useful changes were boring. Cap the context window to what the product actually needs. Use paged attention so fragmentation stops eating a third of the card. Batch by similar sequence length instead of first-come, first-served. And measure tokens per second at a fixed concurrency, not in an empty queue.

People still quote “tokens per second” from a single stream. That number is almost decorative. What matters is tokens per second at the 95th percentile latency your users will tolerate, with a cache already warm, and with a realistic mix of short follow-ups and long pastes.

If you only remember one thing: write down the cache budget first, then pick the quantization, then pick the batch size. Doing it in the other order is how a “small” model ends up needing a second card.