Mixture of Experts
Also called MoE, MoE model, Sparse model
A model design that splits the network into many specialised sub-networks and uses only a few of them for each token, saving compute.
What is mixture of experts?
Mixture of experts (MoE) is a way of building a neural network so that only part of it does the work for any given piece of input. Instead of one large block that processes everything, parts of the model are split into many smaller "experts", and a small router picks a few of them for each token.
Many recent large language models, including several open-weight families, use this design.
How does it work?
Inside each layer of a transformer, the usual single feed-forward block is replaced by many expert blocks plus a router. For each token, the router scores the experts and sends the token to the top one or two. The rest stay idle for that token.
That's why MoE models are described with two numbers: total parameters (all experts together) and active parameters (what's used per token). A model might have hundreds of billions of parameters in total but only a few tens of billions active at a time.
What is it commonly confused with?
- Experts in topics. The experts aren't neatly assigned to subjects like "maths" or "French". They specialise in patterns the model found useful during training, which are rarely easy for people to interpret.
- Model routing. Model routing sends whole requests to different, separate models. MoE routing happens inside one model, token by token.
- A small model. Each token uses fewer parameters, but all of them still have to be loaded into memory.
Why does it matter?
MoE lets a model hold a lot of knowledge while keeping each response fast and relatively cheap to compute, which is a big reason inference costs have fallen. If you want to run one locally, plan memory for the total parameter count, not the active one.
