The six layers of an inference deployment

The GPU itself. Determines what models fit, how fast they run, and most of your hardware budget.

The high-speed wiring inside each chassis that lets multiple GPUs cooperate. Makes tensor parallelism viable.

The network between servers. Optional for single-node replicas, mandatory if a model spans multiple chassis.

Where model weights live before they're loaded, and where cache and telemetry sit at runtime. Affects cold-start time more than steady-state.

The software that actually runs the model. The single largest software lever in your stack.

The facilities-layer answer to whether your rack can handle the heat. Often the real gating constraint, not silicon.