Inside a GPU

In this module, let's take a minute to understand the GPU itself in a bit more detail. Once you know what is inside one, it becomes much easier to see what truly constrains an inference workload and, therefore, what you should be optimizing for.

A GPU is two things in one package

There is a famous demo by NVIDIA's chief, Jensen Huang, that shows how a GPU is fundamentally different from a CPU.

That is probably how you think about a GPU: a device that is chock full of processors that can compute in parallel, accelerating any workload that benefits from heavy parallel processing (image rendering, encoding and decoding, and of course, vector math). However, a modern data-center GPU is two things glued (like, actually glued) together: a set of compute units and a slab of very fast memory that sits right next to those compute units. The two halves are connected by a memory bus that is the widest, fastest connection on the device.

The compute units do the math, while the memory holds the model. The memory bus moves the data between them. If any of the three is too slow, the whole thing becomes too slow.

For inference, the bottleneck is almost always the memory bus.

High Bandwidth Memory: the model's home

The GPU memory that holds the model is called High Bandwidth Memory (HBM). It is a special class of memory designed to do one job extremely well, which is to move enormous amounts of data per second to the compute cores that sit right next to it. It is also very different from the standard DDR5 RAM that has dominated much of computing history.

A few things make HBM different from DDR5 RAM:

• It is physically stacked on top of the GPU package itself, not on a separate stick across the motherboard. This means the wires between the HBM and the cores are millimeters long, not centimeters.
• Because it sits on the same package as the compute cores, its capacity is limited compared to system RAM. A top-tier GPU has 192 GB of HBM.
• The connection is enormously wide. A modern HBM stack has a 1024-bit interface, roughly 16 times wider than DDR5. It is the difference between drinking from a coffee straw and drinking from a firehose (not that I have tried either).

You might wonder if system RAM can help when the model gets too big. The answer is no. System RAM sits behind PCIe, the connection between the CPU and the GPU, and PCIe is dozens of times slower than the HBM bus. Sweeping 70 GB of model parameters across PCIe for every output token would push TPOT (time per output token) from milliseconds into seconds. That is simply not acceptable.

It is the difference between referencing a book on your desk and referencing a book you have to go fetch from the library.

The math of it all

Hopefully you now understand what HBM is and why it is so central. Let's quantify it with some math.

When a model generates its answer, it has to sweep the entire model through the compute units for every single output token (the next module explains why). If the model is 70 GB (which is actually on the smaller end these days) and the memory bandwidth is 3.35 TB/s (an H100), then a single sweep of the model takes about 21 milliseconds. That sets a hard floor on TPOT, regardless of how fast the compute units are.

TPOT floor (ms) = Model size (GB) / Memory bandwidth (GB/s) × 1000

This means a GPU with higher memory bandwidth has a significantly lower floor. On an H200 (4.8 TB/s), the same model's floor drops to 14.6 ms. That improvement does not come from the compute cores. The H200 simply has a wider pipe.

Precision: the first tool in your toolkit

As an infrastructure planner, you are probably trying to stay within a budget. Picture yourself in a room with your AI team, brainstorming ways to meet their latency targets without buying more hardware. What levers do you have?

Your first and most straightforward lever is precision. Suppose the original model is stored at 16 bits (2 bytes) per parameter, a format called BF16 or FP16, and takes up 140 GB of HBM. That is larger than a single H100 (80 GB), so you would need two H100s just for the weights, plus additional logic to split the weights across them. However, if you reduce the precision, meaning the number of bits per parameter, the model gets smaller. At 8 bits (1 byte), the model fits in 70 GB. That format is called FP8 or INT8. At 4 bits, the model is a quarter of its original size.

Cutting precision does three things:

• Reduces the memory footprint. A 70B model at FP8 fits where a 35B model at BF16 would.
• Reduces the bandwidth required per token. Sweeping 70 GB is twice as fast as sweeping 140 GB through the same pipe.
• Costs a small amount of model quality. For most production workloads, FP8 is indistinguishable from BF16. INT4 starts to show quality loss on reasoning-heavy tasks.

The 2026 baseline for production inference is FP8 on Hopper or Blackwell GPUs. If your AI team is still serving at BF16, it is worth asking why.

What comes next

A GPU is compute plus memory, and for inference the memory side usually sets the pace. The next module looks at how a model actually turns a prompt into an answer, and you will see exactly where the compute side and the memory side each take center stage.