ML//Transformer//feed-forward network

The "other half" of each transformer layer: linear → activation → linear. W₂ · ReLU(W₁ · x + b₁) + b₂ (or GELU in modern models)

The "other half" of each transformer layer: linear → activation → linear. W₂ · ReLU(W₁ · x + b₁) + b₂ (or GELU in modern models)

It's basically a perceptron that chunks features: if the current embedding aligns with a feature direction above threshold (ReLU), that feature gets added.

Example: if a vector already points toward "Michael" and "Jordan", the MLP detects compatibility and adds the "basketball" direction. Linears have bias.

Most of the model's parameters live here (2/3), not in attention (1/3). MLP stores facts, attention routes context.

Recent theory: the FFN acts as a key-value memory — keys are the first layer, values the second.

No matter how big the model, the tokenizer (embedding dimension) bottlenecks comprehension — MLP width is limited by it.

Multilayer perceptrons live in MLP layers, not attention layers.