hardware//GPU//GPU kernel//tiling

Tiling is the technique of cutting a large matrix into small square blocks, the tiles, that fit in a processor's fast on-chip memory, so that each block is fetched from slow memory once and reused for many operations before it is discarded. It is how every large matrix product on a GPU is computed, and the reason a GPU can multiply matrices far larger than anything it holds on chip.

Tiling is the technique of cutting a large matrix into small square blocks, the tiles, that fit in a processor's fast on-chip memory, so that each block is fetched from slow memory once and reused for many operations before it is discarded. It is how every large matrix product on a GPU is computed, and the reason a GPU can multiply matrices far larger than anything it holds on chip.

Nothing in a language model fits whole. A 7-billion-parameter model in 16-bit numbers is 14 GB, one of its layers about 400 MB, the query matrix of one layer (4096 × 4096) 33 MB, one head's slice of it (4096 × 128) 1 MB, and a tile of 128 × 128 numbers 32 KB, which finally fits in the 228 KB of shared memory of an H100's SM. So each SM loads a tile, multiplies it by the tokens it holds, adds the result to its partial sums and drops it, while the next tile is already loading. Kernel libraries (cuBLAS, CUTLASS, Triton) do this inside a GPU kernel; model code only writes x @ W.

Hardware use grows with the tokens that reuse each tile.

Fetching a 32 KB tile costs the same whatever follows. When it meets one token, as in a decoding step, it yields 128 × 128 multiply-adds, about 32,000 operations; when it meets 128 tokens at once, as in a prefill, the same 32 KB yield about 4 million. Utilisation climbs with tokens per tile until the ridge of the roofline model (about 295 operations per byte on an H100), where arithmetic takes over as the limit.

Over a whole forward pass the tiles add up to streaming. Every weight crosses from HBM once per step, tile by tile, and the model is never whole on the SMs: 14 GB at 3.35 TB/s is 4.2 ms per step. Somewhat more moves in practice, because some tiles are read again and activations and the KV cache travel too.

CPUs use the same idea, as loop tiling for their caches. On a GPU it is explicit, because shared memory is filled by the program and not by the hardware.

Tile size is a trade-off. Larger tiles reuse each byte more but need more shared memory and registers, so libraries pick sizes per matrix shape and per GPU.