hardware//GPU//architecture

A GPU's architecture is the internal design of the GPU: streaming multiprocessors (NVIDIA) or compute units (AMD), grouped into warps or wavefronts that execute in lockstep. Its memory hierarchy includes registers, shared memory, L1 and L2 caches, and high-bandwidth GDDR or HBM.

A GPU's architecture is the internal design of the GPU: streaming multiprocessors (NVIDIA) or compute units (AMD), grouped into warps or wavefronts that execute in lockstep. Its memory hierarchy includes registers, shared memory, L1 and L2 caches, and high-bandwidth GDDR or HBM.

Architecture choices determine throughput for graphics, ML training and scientific workloads. discrete cards push bleeding-edge architectures, while integrated units share die area with the CPU.

The unit that repeats is the streaming multiprocessor (SM). Each SM holds a set of simple arithmetic lanes, which NVIDIA calls CUDA cores (128 per SM on recent designs), plus tensor cores, specialised units that multiply small matrices in one operation at reduced precision, and its own registers and shared memory. A chip is tens to over a hundred SMs: the H100 in its SXM form enables 132 of them, 16,896 CUDA cores, and the PCIe card 114, 14,592; a consumer RTX 4090 has 16,384.

Core counts do not rank GPUs for AI.

A CUDA core is one floating-point lane, nothing like a CPU core, and the gaming RTX 4090 has almost as many as an H100. What separates them for training is elsewhere: the tensor cores' throughput at low precision (FP16, FP8), the amount and bandwidth of memory (80 GB of HBM at over 3 TB/s against 24 GB of GDDR6X), and the links that join many GPUs into one machine (memory bandwidth, interconnect, roofline model).

The programming model that addresses all of this is CUDA, whose name also labels the lanes; the two meanings are worth keeping apart.