hardware//memory bandwidth

Memory bandwidth is the rate at which data can move between memory and the compute units that consume it, and it decides whether a processor is kept busy or left waiting. A processor with enormous arithmetic capability still waits if operands arrive too slowly.

Memory bandwidth is the rate at which data can move between memory and the compute units that consume it, and it decides whether a processor is kept busy or left waiting. A processor with enormous arithmetic capability still waits if operands arrive too slowly.

Token-by-token LLM inference often streams large weight tensors for relatively little work per byte, making bandwidth a central limit: when serving is dominated by data movement, the bandwidth available caps the aggregate throughput. Reducing precision can therefore accelerate inference by moving fewer bytes even when the abstract number of arithmetic operations changes little.

Compute is the factory. Bandwidth is the road delivering every part. An idle factory may simply have a traffic problem.