ML//mixture of experts

A mixture of experts is a neural network architecture that replaces the single MLP of each layer (the feed-forward network of the transformer block) with \(N\) MLPs in the same position (the experts) and a router that activates only \(K\) of them per token. The total parameter count is huge while the active parameters per token stay small, so the model gains parameters at the same computational cost per token: the scaling trick for frontier models.

A mixture of experts is a neural network architecture that replaces the single MLP of each layer (the feed-forward network of the transformer block) with NNN MLPs in the same position (the experts) and a router that activates only KKK of them per token. The total parameter count is huge while the active parameters per token stay small, so the model gains parameters at the same computational cost per token: the scaling trick for frontier models.

The saving is in arithmetic, and the bill moves elsewhere. All experts must still sit in memory, so a model with 671B parameters needs the memory of a 671B model even when 37B are active; the router must spread tokens evenly or a few experts become the bottleneck while the rest idle (an auxiliary load-balancing objective handles it); and when experts live on different GPUs, every layer sends tokens across the interconnect and back. Total parameters say how much the model can store, active parameters how much each token costs to compute.

It is now the dominant architecture at frontier scale, and dense models are becoming the exception. Mixtral (Mistral) has 8×7B experts with 2 active, 47B parameters in total and 13B active; DeepSeek V3 has 256 experts, and GPT-4 uses MoE.