What is Mixture of Experts (MoE)?

An architecture that activates only part of the model per request, giving large-model quality at lower compute cost.

Mixture of Experts is a model architecture that splits a network's feed-forward layers into many smaller "expert" subnetworks. A lightweight router looks at each token and sends it to only a few experts, so just a fraction of the model's total parameters activate on any given request. This means an MoE model can have a very large total parameter count while its compute cost per token stays close to that of a much smaller dense model. The result is near large-model quality at lower inference cost and higher speed. The catch is memory: all experts must sit in memory even though only some run, so self-hosting still needs substantial hardware. Well-known MoE models include Mixtral, DeepSeek V3 and R1, and several frontier proprietary models are widely believed to use it.

Example

DeepSeek V3 has roughly 671 billion total parameters but activates about 37 billion per token, which is a big reason it undercuts dense models of similar quality on API price.

Why it matters

MoE is why some models offer flagship-level quality at budget prices. If you self-host, remember that a model's "active parameters" figure understates the memory you will actually need. Browse the AI tools directory or the model leaderboard to put it into practice.