ML//mixture of experts//shared expert

A shared expert is an expert of a mixture of experts layer that processes every token, outside the choice of the router, and it is used so that the knowledge every token needs (grammar, formatting, frequent words) is learned once instead of being copied into each routed expert. DeepSeekMoE introduced it together with many small experts; DeepSeek-V3 and Qwen3.6 35B-A3B use one shared expert beside 8 routed experts chosen out of 256.

A shared expert is an expert of a mixture of experts layer that processes every token, outside the choice of the router, and it is used so that the knowledge every token needs (grammar, formatting, frequent words) is learned once instead of being copied into each routed expert. DeepSeekMoE introduced it together with many small experts; DeepSeek-V3 and Qwen3.6 35B-A3B use one shared expert beside 8 routed experts chosen out of 256.

Without it, every routed expert ends up learning the same common material, because any of them may receive any token. With it, that material has one home, and the routed experts are left free to specialise.

Its parameters are active for every token, so it is a fixed cost added to the routed ones: in Qwen3.6 the shared expert of each layer weighs as much as one routed expert, a ninth of the expert compute per token.

The benefit is experimental: ablations in the DeepSeekMoE paper showed lower loss at the same compute with the shared expert than without. That it concentrates common knowledge is the authors' explanation of the result, not something measured directly, and good models exist without one (Mixtral).