hardware//memory-bound

A workload is memory-bound when moving data limits performance before the processor exhausts its arithmetic capacity. Adding more compute units then produces little benefit because they spend more time waiting for operands: for a workload with low arithmetic intensity, the available memory bandwidth sets the ceiling.

A workload is memory-bound when moving data limits performance before the processor exhausts its arithmetic capacity. Adding more compute units then produces little benefit because they spend more time waiting for operands: for a workload with low arithmetic intensity, the available memory bandwidth sets the ceiling.

Autoregressive inference at small batch sizes often has this character: model weights must be read for each generated token with limited reuse. Compression can help by reducing bytes moved, but only if decompression or irregular access does not introduce a new bottleneck.

A workload that waits on a network or a disk rather than on memory is I/O-bound, a different diagnosis with different fixes.

The expensive part of a calculation is sometimes carrying it to the calculator.