Area 01 // LLMsLesson 16 / 14012 min read
Mixture of experts
A mixture-of-experts model replaces one big feed-forward network in each layer with many smaller "expert" networks plus a router that sends each token to only a few of them, so the model can store far more knowledge while doing much less work per token.
Think of it like this
Analogy
A big hospital doesn't send every patient to every doctor. A triage nurse at the front desk looks at each patient and sends them to the two most suitable specialists. The hospital employs a hundred specialists (lots of total knowledge), but each patient only sees two (little time and cost per patient).
In a mixture-of-experts model, the triage nurse is the router, the specialists are the experts, and each patient is a token.
The problem it solves
Bigger models know more and reason better, but in a normal ("dense") transformer every token passes through every weight. Double the parameters and you double the compute for every single token, which doubles cost and latency.
Labs wanted the knowledge of a huge model at the running cost of a much smaller one. Mixture of experts is the main trick that made that possible, and it's why model cards now quote two sizes.
How it works, step by step
- Start from a normal transformer block (lesson 15): attention, then a feed-forward network.
- Replace the single feed-forward network with many experts: separate, smaller feed-forward networks (8, 64, even hundreds).
- Add a router: a tiny layer that gives each token a score for every expert.
- Pick the top-k experts for each token (often 1, 2 or 8) and ignore the rest.
- Run only those experts on that token and blend their outputs, weighted by the router's scores.
- Train everything together, with an extra rule that encourages the router to spread tokens fairly, so no expert sits idle or gets overloaded.
- Result: total parameters (everything stored) can be huge, while active parameters (used per token) stay small.
| Dense model | Mixture-of-experts model | |
|---|---|---|
| Compute per token | All parameters | Only active parameters |
| Memory needed | All parameters | Still all parameters |
| Example sizes (as of November 2026) | 8B dense | Llama 4 Maverick: ~400B total / 17B active; Mistral Large 4: ~1T total / 52B active |
See it in code
- Install
pip install numpy- Needs
- Python 3.10+
import numpy as np
rng = np.random.default_rng(3)
d, n_experts, top_k = 8, 4, 2
experts = [rng.normal(0, 0.3, (d, d)) for _ in range(n_experts)] # 4 small feed-forward nets
router = rng.normal(0, 1.0, (d, n_experts)) # learned: who handles what
def moe_layer(token_vec):
scores = token_vec @ router # one score per expert
chosen = np.argsort(scores)[-top_k:][::-1] # keep only the top-k experts
gate = np.exp(scores[chosen]); gate /= gate.sum() # how much to trust each
out = sum(g * (token_vec @ experts[e]) for g, e in zip(gate, chosen))
return out, chosen, gate
for name in ["token A", "token B", "token C"]:
vec = rng.normal(size=d)
_, chosen, gate = moe_layer(vec)
print(f"{name}: experts {chosen.tolist()} with weights {gate.round(2).tolist()}")
total = n_experts * d * d
active = top_k * d * d
print(f"parameters stored: {total}, used per token: {active} ({active / total:.0%})")token A: experts [1, 3] with weights [0.93, 0.07]
token B: experts [2, 0] with weights [0.97, 0.03]
token C: experts [1, 3] with weights [1.0, 0.0]
parameters stored: 256, used per token: 128 (50%)With 4 experts and top-2, half the expert weights run per token. Real MoE models use many more experts, so often only 5–15% of all parameters are active for each token.
A real example
Example
Self-hosting an MoE model: fast, but memory-hungry
A company wants to self-host an open MoE model with ~400B total and ~17B active parameters. Their first thought: "17B, that fits on one GPU!" It doesn't. All 400B parameters must sit in GPU memory, because any token might need any expert: roughly 800 GB in 16-bit precision, or about 200–250 GB at 4-bit.
Once it's loaded across several GPUs, though, it generates tokens about as fast as a 17B dense model while answering like a much bigger one. They budgeted GPUs for the total size and throughput for the active size.
The easy way
Start here
When you read a model card, look for two numbers: total parameters (decides memory and GPUs) and active parameters (decides speed and cost per token). If you only use models through an API, MoE simply shows up as "fast and cheap for how smart it is".
When you grow: the production setup
Serving MoE well needs expert parallelism (different experts on different GPUs), careful batching so experts stay busy, and quantization to fit total parameters in memory. Serving engines like vLLM support this; it's a big reason GPU sizing questions appear in AI system design interviews.
Trade-offs
What you gain
- Much more knowledge capacity for the same compute per token.
- Faster and cheaper inference than a dense model of the same total size.
- Scales to very large total sizes.
What it costs you
- Needs memory for all experts, so self-hosting is still GPU-heavy.
- Harder to train (router balance, instability) and to serve efficiently.
- Fine-tuning is trickier than for dense models.
When not to use it
- You must run on a single small GPU or a laptop: a small dense model fits; a big MoE won't.
Common mistakes
- Sizing GPU memory by active parameters instead of total parameters.
- Comparing a dense 70B and an MoE "17B active" model as if they were the same size class.
- Assuming each expert is a neat topic specialist you can pick by hand.
- Ignoring that throughput depends on batching many requests so all experts stay busy.
Interview questions
Try answering out loud first, then open the model answer.
What is a mixture-of-experts model and why are so many recent LLMs built this way?
Model answer
It replaces each dense feed-forward layer with several expert networks and a learned router that sends each token to its top-k experts. Total parameters (capacity) grow without proportional growth in per-token compute, since only a few experts run per token. That gives better quality per FLOP, faster inference and cheaper serving at a given capability, which is why many frontier and open models use it.
How do total and active parameters affect deployment?
Model answer
Memory needs scale with total parameters: every expert must be resident (or quickly loadable), so GPUs are sized for the full model, possibly with quantization. Compute and latency per token scale with active parameters plus shared layers, so throughput resembles a much smaller dense model, provided batching keeps experts utilised and expert-parallel communication isn't a bottleneck.
What problems does the router introduce and how are they handled?
Model answer
Routers can collapse onto a few experts (others undertrained) or overload some experts while others idle, which hurts quality and efficiency. Training adds load-balancing auxiliary losses or bias adjustments, capacity limits per expert, and noise in routing. At serving time, uneven routing creates hot spots, so systems balance experts across devices and batch tokens to keep utilisation high.
Quiz
Pick an answer to see whether it’s right and why.
0 of 5 answered
Key terms
- Mixture of experts (MoE)
- A model design where each layer has many expert networks and a router sends each token to only a few of them, so total capacity is large but per-token compute is small.
- Expert (MoE)
- One of the many small feed-forward networks inside a mixture-of-experts layer.
- Router (MoE)
- The small learned layer that scores the experts for each token and picks which ones run.
- Active parameters
- The parameters actually used to process each token in a mixture-of-experts model, as opposed to the total stored.
- Parameter
- Any number a model learns during training (all its weights and biases). Model size is usually quoted in parameters, e.g. 8B = 8 billion.
Before and after
Progress is saved in this browser only.