hardware//memory bandwidth

Memory bandwidth measures how quickly data can move between memory and the compute units that consume it. A processor with enormous arithmetic capability still waits if operands arrive too slowly.

Memory bandwidth measures how quickly data can move between memory and the compute units that consume it. A processor with enormous arithmetic capability still waits if operands arrive too slowly.

Token-by-token LLM inference often streams large weight tensors for relatively little work per byte, making bandwidth a central limit. Reducing precision can therefore accelerate inference by moving fewer bytes even when the abstract number of arithmetic operations changes little.

Compute is the factory. Bandwidth is the road delivering every part. An idle factory may simply have a traffic problem.