hardware//GPU
A GPU is a massively parallel processor designed to apply similar operations across large collections of data. Graphics supplied the original workload; matrix-heavy ML training and inference fit much of the same parallel machinery.
A GPU is a massively parallel processor designed to apply similar operations across large collections of data. Graphics supplied the original workload; matrix-heavy ML training and inference fit much of the same parallel machinery.
It has thousands of simple cores executing the same instruction on different data (SIMD/SIMT). Originally designed for rendering, it is now critical for ML training and inference (the most flexible family of AI accelerator, beside more specialized NPUs, TPUs and ASICs), and it comes as discrete add-in cards or as integrated units sharing the SoC die with the CPU.
Deep learning runs on GPUs because a GPU is, quite literally, a machine for multiplying matrices. A neural network's layers are matrix products, and a product of two large matrices performs on the order of n3n^3n3 operations on n2n^2n2 numbers, so the arithmetic per byte fetched is high and thousands of lanes can be kept busy (roofline model). Training with large batches sits close to the chip's peak; a matrix-vector product, such as one step of a small filter or one token of generation, sits far below it, waiting on memory.
The useful device is larger than its transistor diagram. Drivers, compilers, kernels, libraries and memory systems determine whether an application can reach theoretical throughput, and feeding thousands of arithmetic lanes makes high memory bandwidth central to real performance. In AI, NVIDIA's advantage has historically included this software ecosystem as well as silicon.
In a robot the embedded GPU (Jetson) decides what is ahead (perception, detection, point clouds) and never how much thrust each motor gives. Its average throughput is excellent and its worst-case latency hard to bound, which perception at 30 Hz tolerates and an attitude loop at 500 Hz does not (embedded system).
A GPU wins by making one kind of crowd move together, favouring aggregate work completed over the latency of any single operation. Irregular sequential work still belongs elsewhere.