hardware//GPU//GPU kernel

2026-10-27

A GPU kernel is a function that the CPU launches on a GPU to run as thousands of threads at once, each thread applying the same code to its own part of the data, and it is the unit in which all GPU work is ordered: a matrix product, a softmax or a normalisation each runs as one kernel. The word is unrelated to the OS kernel and to the kernels of kernel methods.

A launch names a grid of thread blocks. The GPU spreads the blocks over its streaming multiprocessors, each block on one SM with that SM's shared memory for its threads, and the kernel ends when the last block finishes. Kernels issued in order on one stream run in that order, each waiting for the one before, so a program is a sequence of fully parallel stages. One layer of a transformer at inference runs as such a sequence, one kernel per stage:

1QQQ, KKK, VVV at oncereads weights2Scores QKTQK^{\mathsf T}QKT3Causal mask and softmax4Weights times VVV5Times WOW_OWO, plus the residualreads weights6LayerNorm7First MLP matrix and activationreads weights8Second MLP matrix, plus the residualreads weights

Inside each stage every token, every head and every output number is computed in parallel; between stages, and between layers, the order is strict. The four stages that read weights carry almost all the memory traffic, and when the model generates one token at a time stages 2 and 4 also read the KV cache.

Every kernel boundary is a round trip to memory.

What one kernel produces and the next one needs is written to HBM and read back, and every launch adds a few microseconds of fixed cost. A step made of many small kernels pays both, which is what kernel fusion removes.

Few people write kernels by hand. Libraries (cuBLAS and CUTLASS from NVIDIA, Triton from OpenAI) and compilers such as torch.compile generate them, and a framework line such as x @ W dispatches one; the language underneath is NVIDIA's CUDA.

Inside a matrix kernel the work is organised as tiling, which decides how much of the hardware the kernel can use.