hardware//GPU//architecture//streaming multiprocessor//shared memory

Shared memory is a small on-chip SRAM inside each streaming multiprocessor of a GPU that the program fills and reads explicitly, visible to all the threads of one block, and it is the workbench where a GPU kernel keeps the piece of data it is about to reuse many times. Nothing enters it on its own, which is what sets it apart from a cache: the kernel copies a tile from HBM into it, makes its threads wait until the copy is complete, computes on it, and overwrites it with the next tile.

Shared memory is a small on-chip SRAM inside each streaming multiprocessor of a GPU that the program fills and reads explicitly, visible to all the threads of one block, and it is the workbench where a GPU kernel keeps the piece of data it is about to reuse many times. Nothing enters it on its own, which is what sets it apart from a cache: the kernel copies a tile from HBM into it, makes its threads wait until the copy is complete, computes on it, and overwrites it with the next tile.

On an H100 each SM has up to 228 KB of it, carved from the same 256 KB of SRAM as its L1 cache, which makes about 30 MB across the 132 SMs. A read from it takes tens of cycles, where a read from HBM takes several hundred. The name says who shares it: the threads of one block, never two processes (the operating-system meaning of the term) nor the CPU and the GPU (unified memory).

Its size decides how a matrix is cut.

A tile of 128 × 128 numbers in 16 bits takes 32 KB, so a few tiles (weights, inputs, partial sums) fit in 228 KB, while one 4096 × 4096 weight matrix (33 MB) never does. Every large matrix product on a GPU is therefore written as tiling, tile by tile through shared memory.

Flash Attention is built on it. Blocks of queries, keys and values are loaded into shared memory and the scores are computed and consumed there, so the full matrix of attention scores is never written to HBM (kernel fusion).

It is one level of the GPU's memory hierarchy, between the registers of each thread and the L2 cache shared by the whole chip.