ML//neural network//weight sharing

Weight sharing is a neural network design technique in which the same parameters are reused at many positions instead of learning separate ones for each, which is why an RNN or LSTM needs one small set of weights for a sequence of any length, and a CNN one small filter for an image of any size.

Weight sharing is a neural network design technique in which the same parameters are reused at many positions instead of learning separate ones for each, which is why an RNN or LSTM needs one small set of weights for a sequence of any length, and a CNN one small filter for an image of any size.

The numbers show what it buys. A 1,000 by 1,000 image has a million inputs; a fully connected layer of 1,000 neurons on it needs a billion weights, and it learns separately to recognize a crack at every position. A convolutional layer with 3 by 3 kernels from 64 to 64 channels has 3⋅3⋅64⋅64+64=36,9283\cdot3\cdot64\cdot64+64=36{,}9283⋅3⋅64⋅64+64=36,928 parameters, whatever the size of the image, and a crack learned in one corner is recognized everywhere.

It bakes in an assumption, an inductive bias: the rule is the same at every position (every timestep, every image patch, every node of a graph). That forces a general rule instead of position-by-position memorization, and makes a CNN translation equivariant, which is right for photos and wrong for spectrograms, where a pattern shifted in frequency means something else.

It is also the core trick of CNNs, where one filter slides across the whole image, and of GNNs, whose matrices are common to all nodes so that the same network serves a fleet of 10 drones or of 200.

It lets a single trained model handle inputs of any length or size without changing shape.