hardware//GPU//architecture//streaming multiprocessor

A streaming multiprocessor (**SM**) is the processor block that an NVIDIA GPU repeats tens or hundreds of times across its die, and it is the unit the GPU hands work to: every block of threads a GPU kernel launches runs on one SM from start to finish. It is the GPU's nearest equivalent of a CPU core (a CUDA core is only one arithmetic lane inside it), and AMD builds the same thing under the name **compute unit**.

A streaming multiprocessor (SM) is the processor block that an NVIDIA GPU repeats tens or hundreds of times across its die, and it is the unit the GPU hands work to: every block of threads a GPU kernel launches runs on one SM from start to finish. It is the GPU's nearest equivalent of a CPU core (a CUDA core is only one arithmetic lane inside it), and AMD builds the same thing under the name compute unit.

Each SM carries its own arithmetic and its own fast memory. On an H100 in its SXM form, which has 132 SMs, one SM holds 128 CUDA cores, four tensor cores for blocks of matrices, a register file of 256 KB and 228 KB of shared memory. Its threads run in groups of 32, the warps, that execute the same instruction together, and the SM switches between warps from one cycle to the next, so that while one warp waits for data another computes.

An SM holds kilobytes and a model holds gigabytes.

The 132 shared memories of an H100 add up to about 30 MB, against 14 GB for a 7-billion-parameter model in 16-bit numbers, so no SM ever holds a model, or even one whole weight matrix. Weights stream through it from HBM in small tiles, each one used and dropped.

Throughput adds up across SMs. The 132 SMs of an H100 reach about 990 TFLOP/s on dense BF16 matrix work together, almost all of it from the tensor cores. The SM count compares GPUs of one family; across families, tensor-core throughput and memory bandwidth say more (GPU architecture).

An SM is only as busy as its data allows. Whether its tensor cores compute or wait depends on how many operations each byte brought from HBM serves, the arithmetic intensity of the roofline model. Generating one token at a time for one user leaves an H100's SMs idle for most of each step (memory-bound).

Where the SM sits among the GPU's memories, and how far each level is from it, is the subject of memory hierarchy.