How a model makes a token

When you send a prompt to an LLM, it produces an answer one token at a time. To do so, two sequential steps need to happen. It is important to understand each one, because each has its own hardware profile and a distinct impact on the infrastructure conversation.

A request lands

When a user sends a question, or really any prompt, the prompt first reaches the inference server. Before it reaches the model, it needs to be converted into tokens. A small piece of software called a tokenizer turns the raw text into tokens. This is fast and happens on the CPU.

Once tokenized, the prompt reaches the GPU, where the language model is already loaded into HBM (that loading happens once, at startup).

Phase 1: Prefill

If you recall the reading sections of the SAT (yes, it may have been a few years), you hopefully read the whole passage and understood its essence before answering any question. The language model's first step, called prefill, is much like you reading the passage for the first time.

In this phase, the GPU reads the incoming prompt. However, instead of reading the prompt one token at a time, the GPU reads all of it in parallel, much like an F1 pit crew changing all four tires at once. The thousands of compute cores inside the GPU work together to process the entire prompt in a single pass.

The output of the prefill phase is something called the key-value (KV) cache, which you can think of as the model's working memory. It is your understanding of the passage, which you will rely on as you answer the questions. The KV cache gets its own module shortly.

Phase 2: Decode

Once the model understands the prompt, it generates the response one token at a time. Here it is much more like us, stringing one word after another. This phase is called decode.

To generate a single token, the GPU has to sweep through the entire 70 GB of model parameters. Then it repeats that sweep for every token after that. Every output token requires the entire model to flow through the GPU's memory bus, end to end.

What does this mean for the infrastructure?

Prefill, where the entire prompt is read at once, is bound by the GPU's raw compute power. The strain on the compute cores is proportional to the length of the prompt: the longer the prompt, the greater the strain.

Decode is different. The GPU is constantly moving 70 GB of data per token, and while it does that, the massive compute units sit mostly idle, waiting for data to arrive from memory. That is the memory bandwidth floor from the previous module in action. Decode is bound by memory bandwidth, not compute.

Why should I care?

Earlier in the curriculum, we briefly met the two key measures of latency:

• TTFT, Time To First Token. This is the pause before your ChatGPT or Claude window shows even a single word. That delay is set by the prefill phase.
• TPOT, Time Per Output Token. This is the time the model takes to generate each subsequent token. As you probably guessed, the decode phase sets it.

Both shape how fast your users perceive the system to be. So the infrastructure you use for inference needs to serve both the prefill and decode phases well.

An example

The total time a user waits is TTFT plus the number of output tokens multiplied by TPOT. A 300-token answer at 30 ms per token, plus a 200 ms prefill, takes about 9.2 seconds. Streamed back token by token, that feels natural. Delivered as a single block at the end, it would feel slow.

What this means for IT planners

As might be painfully obvious by now, prefill and decode are bound by two completely separate GPU characteristics, each expensive in its own right. That poses an interesting challenge for you as the infrastructure planner. Do you have your cake or do you eat it?

The answer depends on the nature of the workloads your AI team expects:

• Long prompts and short answers. Workloads like analyzing legal contracts or financial logs spend most of their time in prefill. You need to provision for heavy GPU compute.
• Short prompts and long answers. Interactive chatbots and code generation assistants spend most of their time in decode. You need to prioritize memory bandwidth.

Summary and what comes next

In this module, we looked at prefill and decode, the two sequential steps that take an input prompt and give you... a "princess." Umm, nope. They give you an output. We also saw that the two steps are like oil and water in their GPU requirements. Prefill is bound by the GPU's compute cores, while decode is constrained by the bandwidth of the GPU's memory.

Prefill also left something behind: the KV cache. The next module looks at what lives inside it, and why it ends up deciding how many users your infrastructure can serve at once.