ML//transformer//feed-forward network

The feed-forward network is the other half of each transformer layer, the one that stores and transforms what attention routes between tokens: a linear map, an activation and another linear map, \(W_2 \operatorname{ReLU}(W_1 x + b_1) + b_2\) (with GELU in place of ReLU in modern models). Both linear maps have a bias.

The feed-forward network is the other half of each transformer layer, the one that stores and transforms what attention routes between tokens: a linear map, an activation and another linear map, W2ReLU⁡(W1x+b1)+b2W_2 \operatorname{ReLU}(W_1 x + b_1) + b_2W2​ReLU(W1​x+b1​)+b2​ (with GELU in place of ReLU in modern models). Both linear maps have a bias.

It is basically a perceptron that chunks features: if the current embedding aligns with a feature direction above threshold (ReLU), that feature gets added. If a vector already points toward Michael and Jordan, for instance, the MLP detects the compatibility and adds the basketball direction.

Attention mixes information across positions; the FFN works on each position on its own, the same weights applied to every token. It is where the layer's nonlinearity enters, and the residual connections and normalization around both halves keep a deep stack trainable.

Most of the model's parameters live here (about two thirds), and the rest in attention (about one third).

A recent line of theory reads the FFN as a key-value memory, with the keys in the first layer and the values in the second.

No matter how big the model, the tokenizer (embedding dimension) bottlenecks comprehension. MLP width is limited by it.

Multilayer perceptrons live in MLP layers, not attention layers.