ML//mixture of experts//router

The router is the gating mechanism in Mixture of Experts that decides which experts process each token. It is a small neural network (often a single linear layer followed by a softmax) that takes a token's representation and outputs a probability distribution over the experts.

The router is the gating mechanism in Mixture of Experts that decides which experts process each token. It is a small neural network (often a single linear layer followed by a softmax) that takes a token's representation and outputs a probability distribution over the experts.

It typically routes each token to the top one or two experts out of 8 to 64 or more. That is what makes MoE efficient: only a fraction of the parameters are active per token, so a 100B parameter MoE model might use only 13B parameters per forward pass.

Without constraints the router tends to collapse, sending most tokens to the same one or two experts while the others go unused, which wastes capacity and defeats the purpose. The fix is an auxiliary loss that penalizes uneven expert utilization.

Routing changes the scaling laws, because MoE decouples the parameter count from the compute cost. Parameters can grow (more experts) without the FLOPs growing in proportion (each token still hits the same number of experts), which changes the optimal scaling relationship.

Token-level routing means different tokens in the same sequence can hit different experts. The model implicitly learns to specialize experts: some handle code, others handle math, others handle natural language. But this specialization is emergent, not designed.

The feed-forward network layers are what get replicated as experts. Attention is shared across all tokens (global context), but FFN computation is routed (specialized processing). This split reflects the observation that attention handles where to look while the FFN handles what to do with it.