AI Explainedby Milad Fahmy

AI Explained

Mixture of Experts (MoE)

DeepSeek V3 has 671 billion parameters, but each token uses only 37 billion. How Mixture of Experts models route tokens to a few experts, and the trade-offs.

Milad FahmyNov 1, 20262 min read

On this page
  1. Dense vs mixture of experts
  2. A concrete example: Mixtral
  3. The experts aren't topic experts
  4. Training has a catch
  5. Bigger and bigger
  6. The trade-off
  7. Reading model specs

DeepSeek V3 has 671 billion parameters. But each token it processes only uses 37 billion of them. How? It's a Mixture of Experts.

Dense vs mixture of experts

In a normal dense model, every parameter works on every token. A bigger model means more compute, every single time.

In a mixture of experts (MoE) model, some layers are split into many experts: small feed-forward networks side by side. A router looks at each token and sends it to just a few experts, often two. The rest sit idle for that token.

A concrete example: Mixtral

Mixtral 8x7B has eight experts per layer and picks two for every token. It has 47 billion parameters in total but uses about 13 billion per token. Its authors reported it matched or beat Llama 2 70B and GPT-3.5 on the benchmarks they tested: more knowledge, for the compute cost of a much smaller model.

The experts aren't topic experts

It's tempting to imagine a math expert and a biology expert. In practice, Mixtral's authors found no clear pattern of experts specializing by topic. The router learns whatever split works.

Training has a catch

The router can learn to favor a few experts and ignore the rest. So MoE models add load balancing during training to spread the work. DeepSeek V3, for example, uses its own auxiliary-loss-free balancing strategy.

Bigger and bigger

Meta's Llama 4 Maverick has 128 routed experts plus a shared expert that every token visits: 400 billion parameters in total, 17 billion active. The idea isn't new, though: a 2017 Google paper (Shazeer et al.) used up to thousands of experts for over a thousand times more capacity.

The trade-off

All the experts must sit in memory, even if only a few run per token. So MoE models are fast per token but heavy to host.

Reading model specs

When you see two numbers, total and active:

  • Active parameters tell you the speed and compute cost per token.
  • Total parameters tell you how much memory you need.

Many experts, a few at a time.

Sources

#mixture of experts#moe#llm architecture#open models#inference

Watch