ML//NPU

An NPU is a dedicated accelerator for ML inference workloads, optimized for matrix multiply-accumulate at low precision (INT8, FP16). It offloads inference from the CPU and GPU and achieves higher throughput per watt. Like the TPU it is an ASIC for tensor multiplication, but it runs only the forward pass, quantized, while the TPU also runs backpropagation for training at higher precision.

An NPU is a dedicated accelerator for ML inference workloads, optimized for matrix multiply-accumulate at low precision (INT8, FP16). It offloads inference from the CPU and GPU and achieves higher throughput per watt. Like the TPU it is an ASIC for tensor multiplication, but it runs only the forward pass, quantized, while the TPU also runs backpropagation for training at higher precision.

Its size class is one to tens of TOPS (trillions of operations per second) at a few watts, integrated into an embedded processor or sold as a separate chip. That is the middle tier of edge inference: above a microcontroller running a network of a few hundred KB in milliwatts (TinyML), below an embedded GPU such as a Jetson that spends tens of watts to run CNNs and mid-size transformers at tens of images per second. Vision in a smart camera on an inspection line, or object detection on a drone, is typical NPU work, with the network quantized to int8 first (quantization).

NPUs are found inside modern SoC designs (Apple Neural Engine, Qualcomm Hexagon, Google Tensor) and are programmed through frameworks such as Core ML, NNAPI or ONNX Runtime. The NPU is passive, waking the MCU only when it detects a pattern, and it is used for continuous events (sound) where always-on sensor filtering is not enough. Its practical limit is the operator list: a layer the NPU's compiler does not support falls back to the CPU, and the latency measured on the bench changes.