Mixture of Experts
Expanded definition
Mixture of experts, or MoE, is a neural-network architecture with multiple expert components and a router that chooses which experts process each token. Sparse activation can increase total model capacity without using every parameter for every token. Total parameters and active parameters therefore describe different things. A model may have a very large total parameter count while activating a smaller subset during inference. This can improve compute efficiency, but it does not make serving automatically cheap or simple: weights still need storage and memory, routing must be balanced, and distributed communication can become a bottleneck. MoE is an architecture choice, not a quality guarantee. Teams should evaluate the exact model on task quality, latency, throughput, memory, and serving reliability.
Mixture of experts vs dense model
| Dimension | Mixture of experts | Dense model |
|---|---|---|
| Token path | Router selects a subset of experts | Each token uses the same dense blocks |
| Parameter reporting | Total and active parameters differ | Most model parameters participate in each pass |
| Serving challenge | Routing, memory placement, and communication | Large matrix compute and memory bandwidth |
| Quality | Depends on training and routing quality | Depends on training and architecture quality |
MoE can provide more total capacity per active-token compute, but production value depends on the exact model and serving stack.
Related terms
Explore adjacent ideas in the knowledge graph.
Mixture of Experts FAQ
What is Mixture of Experts?
A mixture-of-experts model routes each token through a selected subset of specialized neural-network components instead of activating the full parameter set.
How is Mixture of Experts used in AI systems?
Mixture of experts, or MoE, is a neural-network architecture with multiple expert components and a router that chooses which experts process each token. Sparse activation can increase total model capacity without using every parameter for every token. Total parameters and active parameters therefore describe different things. A model may have a very large total parameter count while activating a...
Related
Comparisons, tools, and models that connect to this idea.