Mixture of experts
Expanded definition
MoE architectures pair a router with multiple expert feed-forward blocks. Sparse activation can raise total parameter count without using every parameter on every token, which is how some frontier open-weight models advertise large totals with smaller active counts. Serving still needs capacity for the experts you load, plus routing overhead. MoE is not a guarantee of quality, and expert-parallel serving is operationally different from a dense model of the same active size. Check the vendor card for active parameters, not only the headline total.
Mixture of experts vs dense model
| Dimension | Mixture of experts | Dense model |
|---|---|---|
| Token path | Router selects a subset of experts | Each token uses the same dense blocks |
| Parameter reporting | Total and active parameters differ | Most model parameters participate in each pass |
| Serving challenge | Routing, memory placement, and communication | Large matrix compute and memory bandwidth |
| Quality | Depends on training and routing quality | Depends on training and architecture quality |
MoE can provide more total capacity per active-token compute, but production value depends on the exact model and serving stack.
Related terms
Explore adjacent ideas in the knowledge graph.
Mixture of experts FAQ
What is Mixture of experts?
A mixture-of-experts model routes each token through a subset of specialized networks instead of activating every parameter.
How is Mixture of experts used in AI systems?
MoE architectures pair a router with multiple expert feed-forward blocks. Sparse activation can raise total parameter count without using every parameter on every token, which is how some frontier open-weight models advertise large totals with smaller active counts. Serving still needs capacity for the experts you load, plus routing overhead.
Related
Comparisons, tools, and models that connect to this idea.