ML//model compression//model pruning

Model pruning is a model compression technique that removes weights, neurons, attention heads or larger structures estimated to contribute little to a model's useful behavior. The result may be unstructured sparsity (individual zeros) or structured sparsity (whole blocks that regular hardware can skip more easily).

Model pruning is a model compression technique that removes weights, neurons, attention heads or larger structures estimated to contribute little to a model's useful behavior. The result may be unstructured sparsity (individual zeros) or structured sparsity (whole blocks that regular hardware can skip more easily).

The simplest version removes the weights closest to zero, on the reasoning that a near-zero weight barely changes any output, and then fine-tunes the network briefly so the remaining weights absorb the loss. On edge hardware it is one tool of a box of three, beside quantization (fewer bits per weight) and knowledge distillation (a small network trained to imitate a large one); quantization is usually the first and the cheapest.

Parameter count can fall without wall-clock inference improving. If the runtime still loads dense tensors or sparse indexing costs more than it saves, the compression remains mathematical rather than operational: on memory-bound inference, pruning helps only when it reduces the bytes actually transferred. Structured pruning often sacrifices some freedom to buy hardware legibility, which is why a microcontroller or an NPU usually benefits from removing whole channels and seldom from scattered zeros.

Deleting multiplication is easy. Convincing the accelerator not to wait for it is the engineering.