ML//transformer//feed-forward network//SwiGLU

SwiGLU is a form of the transformer's feed-forward network in which one projection of the token, passed through the SiLU activation, multiplies element by element a second projection of the same token before the output projection, and it is used in PaLM, LLaMA and most recent open models because it gives better quality than a plain GELU block for the same number of parameters. The second projection acts as a gate: each hidden unit's value is scaled up or switched off by a number that also depends on the input.

SwiGLU is a form of the transformer's feed-forward network in which one projection of the token, passed through the SiLU activation, multiplies element by element a second projection of the same token before the output projection, and it is used in PaLM, LLaMA and most recent open models because it gives better quality than a plain GELU block for the same number of parameters. The second projection acts as a gate: each hidden unit's value is scaled up or switched off by a number that also depends on the input.

FFN(x)=(SiLU(xW1)⊙xW3) W2,SiLU(z)=z σ(z)\text{FFN}(x)=\big(\text{SiLU}(xW_1)\odot xW_3\big)\,W_2,\qquad \text{SiLU}(z)=z\,\sigma(z)FFN(x)=(SiLU(xW1​)⊙xW3​)W2​,SiLU(z)=zσ(z)

The block therefore has three matrices instead of two: W1W_1W1​ and W3W_3W3​ go from the model width ddd to the hidden width, W2W_2W2​ comes back. σ\sigmaσ is the sigmoid.

To keep the parameter count of the classic block (two matrices of d×4dd\times4dd×4d, so 8d28d^28d2), the hidden width is cut to about two thirds of 4d4d4d: three matrices of d×83dd\times\tfrac{8}{3}dd×38​d are again 8d28d^28d2. LLaMA 7B uses d=4096d=4096d=4096 and a hidden width of 11,008, close to 83⋅4096\tfrac{8}{3}\cdot409638​⋅4096 rounded up to a multiple of 256.

The gain is measured, not derived: the paper that introduced it compared several gated variants on the same budget and reported lower loss, without a theory of why. Read the third matrix in a model file as the sign of this block, not as a bigger feed-forward network.

In a mixture of experts each expert is usually one of these blocks, which is why a file lists three matrices per expert.