Dictionary term
Mixture of Experts
Also called MoE · sparse model
A mixture-of-experts (MoE) model contains many expert sub-networks and a router (gate) that sends each token to only a few of them, so the model can have a very large total parameter count while using only a fraction of it per token.
In plain English
An MoE model is a big AI made of many specialist parts, and only the few parts needed for each word switch on, which keeps it faster and cheaper to run.
Analogy
Like a large consultancy where each question goes to two or three relevant specialists instead of the whole firm.
Why it matters
Most large open-weight releases now use MoE, and their names often list both total and active parameters. Active parameters drive speed and cost per token, but you still need memory for all the weights, which matters when sizing self-hosted hardware.
In practice
Moonshot says its Kimi K3 has 2.8 trillion parameters but activates 16 of its 896 experts for each token.
Related terms
Sources
Browse all terms →Last verified