hardware//GPU//architecture//streaming multiprocessor//tensor core

A tensor core is an arithmetic unit inside each streaming multiprocessor of an NVIDIA GPU that multiplies a small block of one matrix by a block of another and adds the result to an accumulator in a single instruction, and it is where almost all the arithmetic of training and running a neural network is done. AMD's equivalent is the **matrix core**. Despite the name, it works on two-dimensional blocks; a tensor of higher rank reaches it cut into matrices.

A tensor core is an arithmetic unit inside each streaming multiprocessor of an NVIDIA GPU that multiplies a small block of one matrix by a block of another and adds the result to an accumulator in a single instruction, and it is where almost all the arithmetic of training and running a neural network is done. AMD's equivalent is the matrix core. Despite the name, it works on two-dimensional blocks; a tensor of higher rank reaches it cut into matrices.

A CUDA core performs one multiply-add on one pair of numbers per cycle. A tensor core performs a whole block product, hundreds of multiply-adds, in the same instruction, at reduced precision (FP16, BF16, FP8, INT8) and with the sums kept in higher precision. An H100 has four per SM, 528 in all, and they deliver about 990 TFLOP/s of dense BF16, where its 16,896 CUDA cores give about 67 TFLOP/s in FP32. Each halving of the bits roughly doubles the rate: about 1,980 TFLOP/s in FP8.

A tensor core is fast only when it is fed.

At 990 TFLOP/s against 3.35 TB/s of memory bandwidth, an H100 needs about 295 operations for every byte it brings from HBM to keep its tensor cores busy (the ridge of the roofline model). One token multiplied by a weight matrix gives about one operation per byte, so the units wait; a matrix of many tokens, cut into tiles that each serve all of them, keeps them working.

Code rarely names them. Kernel libraries (cuBLAS, CUTLASS, Triton) emit the tensor-core instructions, and a framework line such as x @ W on BF16 tensors runs on them through those libraries (GPU kernel).

Their number is not the number to compare between GPUs: the throughput per precision is, together with memory bandwidth (GPU architecture).