hardware//GPU//VRAM

VRAM is the memory soldered on a graphics card next to its GPU and wired to it alone over a very wide bus, and in local inference it is the memory a model must fit in to run at the card's full speed. Consumer cards carry GDDR (16 to 32 GB on current NVIDIA models); data-centre accelerators carry HBM (80 GB on an H100).

VRAM is the memory soldered on a graphics card next to its GPU and wired to it alone over a very wide bus, and in local inference it is the memory a model must fit in to run at the card's full speed. Consumer cards carry GDDR (16 to 32 GB on current NVIDIA models); data-centre accelerators carry HBM (80 GB on an H100).

Its bandwidth is what makes a discrete GPU fast at generation: an RTX 5090 reads its 32 GB at 1,792 GB/s and an H100 its 80 GB at 3,350 GB/s, against 128 GB/s for a thin laptop's unified memory. Since every generated token reads every active weight, a model held in VRAM writes many times faster.

VRAM is small and fast, system RAM is large and slow, and PCIe joins them (about 64 GB/s on a PCIe 5.0 x16 slot). A model that does not fit in VRAM is split: the layers that fit run on the GPU, the rest on the CPU from system RAM, and the slow part sets the pace.

What must fit is more than the weights: the KV cache for the reserved context and the compute buffers too (model memory footprint).

ollama ps shows the split in its PROCESSOR column: 100% GPU when everything fits, a CPU and GPU percentage when it does not.

The PCIe link copies the model into VRAM once at load time, not once per token, so its speed matters for loading and for split models only.