hardware//GPU//GPU kernel//kernel fusion

Kernel fusion is the merging of several consecutive GPU operations into one GPU kernel, so that the intermediate results stay in registers and shared memory instead of being written to HBM and read back, and it is the main way to speed up the parts of a model that are memory-bound without changing their arithmetic. Each kernel boundary costs a write, a read and a launch; fusing removes all three.

Kernel fusion is the merging of several consecutive GPU operations into one GPU kernel, so that the intermediate results stay in registers and shared memory instead of being written to HBM and read back, and it is the main way to speed up the parts of a model that are memory-bound without changing their arithmetic. Each kernel boundary costs a write, a read and a launch; fusing removes all three.

The simplest case is an activation after a matrix product. Unfused, the product writes its whole output to HBM, and a second kernel reads it, applies the activation and writes it again. Fused, the activation is applied to each tile of the output before it leaves the SM, and the tensor crosses the memory bus once.

Fusion changes where numbers live, not what they are.

The mathematics is the same and the outputs match up to the order of rounding. Flash Attention fuses the scores QKTQK^{\mathsf T}QKT, the mask with the softmax, and the weighted sum of VVV into one kernel, so the n×nn \times nn×n matrix of scores is never written to HBM; an online softmax, which rescales its running sum as each new block of scores arrives, keeps the result exact.

It is done by hand or by a compiler. The heaviest cases (attention, normalisation layers) have hand-written fused kernels in inference engines; chains of element-wise operations are fused automatically by torch.compile, which generates Triton kernels, and by XLA for JAX.

It saves bytes, not operations. A chain of light operations on a large tensor gains a lot; a large matrix product that is already compute-bound gains almost nothing, and two such products in a row are usually left as separate kernels.