hardware//SIMD

SIMD (**single instruction, multiple data**) is a way of building a processor in which one decoded instruction drives many arithmetic lanes at once, each lane applying the same operation to its own number, and it is how CPUs and GPUs get through long vectors and matrices without paying for one instruction per number. A dot product of 4,096 elements, the basic operation of a neural network, is the typical case: the same multiply and add, repeated over data that does not depend on itself.

SIMD (single instruction, multiple data) is a way of building a processor in which one decoded instruction drives many arithmetic lanes at once, each lane applying the same operation to its own number, and it is how CPUs and GPUs get through long vectors and matrices without paying for one instruction per number. A dot product of 4,096 elements, the basic operation of a neural network, is the typical case: the same multiply and add, repeated over data that does not depend on itself.

The price of an instruction is mostly control: fetching it, decoding it, predicting branches, keeping caches that feed it. SIMD pays that price once and spreads it across all the lanes. A CPU core does it with vector registers: with AVX-512 (Zen 5, recent Intel server chips) one register holds 512 bits, 16 numbers in 32-bit float, and one instruction multiplies all 16. A GPU takes the idea much further: one instruction drives 32 lanes of an NVIDIA warp or up to 64 lanes of an AMD compute unit, and NVIDIA calls its variant SIMT because each lane looks to the programmer like its own thread (streaming multiprocessor).

A CPU core is a small independent computer that runs its own instruction stream fast, with large caches and branch prediction; a GPU lane is only an arithmetic unit that does what the shared decoder tells it. The GPU spends its silicon on lanes instead of control, which is why it wins on uniform matrix work and loses on branchy code.

Branches are the limit. When lanes that share an instruction disagree on an if, the hardware runs both paths one after the other with the inactive lanes masked off (divergence), so a branch that splits a group in half halves its throughput.

Lanes are not workers. A compute unit or an SM is one block of lanes behind one decoder, not 64 independent agents, and counting lanes compares chips only when the work keeps every lane busy on the same instruction.

The arithmetic of a transformer is almost entirely matrix multiplication, so it maps onto SIMD lanes and onto tensor cores, which are matrix units built on the same principle; how fast the lanes can be fed is a separate question of memory bandwidth.