The KV cache, the silent capacity killer

The last module ended with HBM holding three things: the model weights, the KV cache, and a small overhead. The model weights you already understand. This module covers the second tenant, the KV cache, and why it ends up being the single biggest determinant of how many users your infrastructure can serve.

If you take one thing from this module, take this: the KV cache grows in two directions simultaneously, with context length and with concurrent users, and the two multiply. When you change either one, your memory budget moves by an order of magnitude, not a small percentage.

What actually lives in the KV cache

When a model reads a prompt during the prefill phase, it computes intermediate values for every single token. The model could throw these values away and recompute them on every single step of the decode loop, but that would be wildly expensive and tank performance. The math dictates a smarter path: store them and reuse them.

These stored intermediate values are the KV cache. The name comes from the math of attention, the core mechanism that lets each token look back at every other token in a sequence. This is how the model contextualizes words. The model understands that the word "bank" means a financial institution in one sentence or a river edge in another, based entirely on its relationship to the surrounding words. For every token in a conversation, the model stores two vectors, called the key and the value, directly in the GPU's high-bandwidth memory (HBM).

Two non-negotiable infrastructure facts govern this storage:

1. The storage footprint per token is fixed by the model's architecture. A given model uses a specific number of bytes per token, regardless of what text or characters are being processed. This remains true for even modern Mixture-of-Experts (MoE) models. While the routing "experts" switch out dynamically, the underlying attention head architecture still dictates a fixed memory cost per token.
2. That storage must live exclusively in ultra-expensive HBM, right alongside the model weights themselves. Every active conversation's memory is aggressively competing for the exact same physical VRAM footprint as the model itself.

Deconstructing the math: a Llama 3.1 70B sandbox

To see how these numbers materialize on a spec sheet, let's open the hood of a standard enterprise baseline: Llama 3.1 70B running at FP8 precision (8-bit floating point, or 1 byte of data per parameter).

The physical size of a single token's KV cache is calculated using a fixed architectural formula:

KV bytes per token = 2 × Layers × KV heads × Head dimension × Bytes per element

For Llama 3.1 70B, the hardware spec sheet lists the following constants:

• Layers: 80
• Key-value heads: 8 (using Grouped-Query Attention)
• Head dimension: 128
• Bytes per element: 1 byte (due to FP8 precision)

When you multiply it out:

2 × 80 × 8 × 128 × 1 = 163,840 bytes ≈ 160 KB per token

How it scales

If one single token requires 160 kilobytes of dedicated VRAM, a standard 1,000-token conversation consumes about 160 megabytes. That feels small, but look at what happens as context lengths expand to modern enterprise requirements:

• A 10,000-token document interaction takes 1.6 gigabytes.
• A 128,000-token context window, the maximum supported by modern models, takes roughly 40 gigabytes of memory. And that is for one single user.

Now multiply by concurrency. It is easy to focus only on model size when building a cluster footprint. A team picks a 70B model, multiplies 70 GB by an operational buffer, and feels reasonably confident. But the moment 5 corporate users upload massive document archives totaling 128,000 tokens simultaneously, you suddenly need 200 gigabytes of KV cache space, plus 70 gigabytes for the frozen model weights. You have just rocketed past the 160 GB memory limit of two combined NVIDIA H100 GPUs.

The reality is a strict equation where context and concurrency multiply:

Total VRAM needed = Model size + (Context length × Concurrent users × Bytes per token)

A concrete budget

Picture a 70B model at FP8 serving an enterprise Retrieval-Augmented Generation (RAG) application:

• Average prompt: 6,000 tokens of retrieved internal documentation and chat history.
• Peak concurrency: 64 active, in-flight requests processing simultaneously.

At 6,000 tokens, each individual user's KV cache takes up roughly 1 gigabyte of VRAM. Multiply that by 64 concurrent users, and the engine requires a hard baseline of 64 GB of memory exclusively for user state.

The physical hardware budget breaks down like this:

• Model weights (FP8): 70 GB
• KV cache (64 users × 6K context): 64 GB
• Headroom, activations, and overhead: ~46 GB
• Total VRAM required: ~180 GB

This workload fits comfortably across two H200s (282 GB combined) or two AMD MI300X cards. It will fail to load entirely on two standard H100s (160 GB combined).

The exact same workload, running on the exact same model, will land on completely different hardware tiers based on a single parameter: context length. If you halve the context to 3,000 tokens, the KV cache drops to 32 GB, allowing the entire system to fit neatly onto two standard H100s. If you double the context to 12,000 tokens, the KV cache swells to 128 GB. Suddenly, you need to purchase a third H200 card, implement separate quantization recipes, or have a direct conversation with the product team about whether a 12,000-token history window is actually an absolute business requirement.

Why this is the silent killer

The KV cache remains a silent killer because it rarely appears on standard software intake forms. The AI development team rarely calculates it explicitly because their serving runtime abstracts it away. The infrastructure team has no true equivalent to this behavior in traditional software architectures. As a result, the compounding multiplication of context length and concurrency slips right through the organizational gap.

To fix this, you must mandate three explicit questions before sizing any inference cluster:

1. What is the hard maximum context length the application will accept?
2. What is the realistic, absolute peak of concurrent, in-flight requests?
3. What specific precision (e.g., FP8) will the KV cache be stored in?

Multiplying those three numbers gives you your honest KV memory budget. Add the base model weights, and you have your true physical infrastructure floor.

What comes next

The next module covers three architectural levers that reduce KV pressure without buying more GPUs. After that, Part II closes by revisiting the four assumptions that traditional infrastructure planning makes about AI workloads, all of which now break for reasons you can name.

Try this in the SizerOpen the Sizer with a 128K-token context and watch the memory budget swell. Then drop max_context_tokens to 8,000 and see the recommendation collapse to fewer GPUs. This is what context length pressure looks like in practice.