hardware//memory hierarchy

A memory hierarchy is the arrangement of a computer's memories in levels, from small, fast and close to the arithmetic to large, slow and far from it, and it exists so that a processor can work at the speed of its fastest level while holding data at the price of its cheapest. Every machine has one: on a laptop CPU it runs from registers through the caches to RAM and storage. On a GPU server running a language model it is a ladder that every weight climbs down:

A memory hierarchy is the arrangement of a computer's memories in levels, from small, fast and close to the arithmetic to large, slow and far from it, and it exists so that a processor can work at the speed of its fastest level while holding data at the price of its cheapest. Every machine has one: on a laptop CPU it runs from registers through the caches to RAM and storage. On a GPU server running a language model it is a ladder that every weight climbs down:

Level Size on an H100 server Link to the next level

NVMe SSD terabytes about 7 GB/s

Host RAM 64 to 512 GB PCIe 5.0 x16, about 64 GB/s

HBM (the GPU's VRAM) 80 GB memory bus, 3.35 TB/s

L2 cache 50 MB on chip

Shared memory 228 KB per SM, about 30 MB on chip

Registers 256 KB per SM, about 34 MB inside the SM

Moving data down a level takes its size divided by the bandwidth of the link. For a 14 GB model (7 billion weights in 16 bits) that is about 2 s from SSD to RAM and 220 ms from RAM to HBM, both paid once when the model loads; a model counts as loaded when it sits in HBM. From HBM to the SMs it is 4.2 ms, and that one is paid at every step.

The level a workload reads from on every step sets its speed.

A CPU program that reuses the same data runs at cache speed. A language model generating text reuses nothing within a step: each weight is streamed once from HBM in tiles and dropped, and 14 GB cannot stay in a 50 MB L2 cache, so decoding runs at HBM bandwidth (memory-bound). Keeping the weights one level higher, in host RAM across PCIe, would make it about fifty times slower.

The bandwidth belongs to the link, not to the memory. HBM is the memory; the 3.35 TB/s is the bus between it and the chip, and the same 80 GB on a narrower bus would be slower.

Capacity decides what fits and bandwidth decides how fast it runs: a model that fits in host RAM but not in HBM runs, slowly (model memory footprint).