hardware//RAM//HBM

High-bandwidth memory (HBM) is DRAM built as a vertical stack of dies connected by through-silicon vias and placed next to the processor on a silicon interposer, and it is the memory that feeds AI accelerators, whose speed is limited more by moving data than by arithmetic. Each stack talks to the processor over a very wide interface (1024 bits for HBM3, 2048 for HBM4), which is what gives it its bandwidth.

High-bandwidth memory (HBM) is DRAM built as a vertical stack of dies connected by through-silicon vias and placed next to the processor on a silicon interposer, and it is the memory that feeds AI accelerators, whose speed is limited more by moving data than by arithmetic. Each stack talks to the processor over a very wide interface (1024 bits for HBM3, 2048 for HBM4), which is what gives it its bandwidth.

A conventional memory module sits centimetres away on a board and connects through a narrow bus; HBM sits millimetres away and connects through thousands of wires in the package (advanced packaging). That is why a large model, which is memory-bound when it generates text, depends on HBM as much as on the processor.

Three companies make it (SK Hynix, Samsung and Micron), and output is sold under agreements a year or more in advance, so HBM supply has been one of the inputs that limit how many accelerators can be built.

The price of the bandwidth is cost and heat: stacked dies are harder to make and to cool than ordinary memory.