hardware//model memory footprint

Model memory footprint is the device memory required to run a workload, often allocated in GPU VRAM. It includes weights, activations, KV cache, temporary workspaces and allocator overhead.

Model memory footprint is the device memory required to run a workload, often allocated in GPU VRAM. It includes weights, activations, KV cache, temporary workspaces and allocator overhead.

This is why weights that fit in memory do not guarantee that inference fits. Batch size, context length, numerical precision and serving strategy change the remaining components. Compression may halve weight storage while total memory falls by much less when KV cache dominates: weight quantization shrinks one major component of the footprint and leaves the other allocations of inference as they were.

Training needs far more memory than inference for the same model. Backpropagation computes every gradient in a pass costing only two or three forward passes, but to do so it must keep the activations of every layer from the forward pass until the backward pass reaches them; the optimizer adds its own state per weight on top. Inference can discard each layer's activations as soon as the next layer has used them. So a network trained on a data-centre GPU can run on a small edge device, and the reverse is rarely true (edge computing).

VRAM is not a cupboard containing only the model. It is the workbench, the parts tray and the notebook at once.