Mixture of Experts: The Architecture Trick Behind Every Modern AI Model
Here's an open secret about the most powerful AI models in the world: they don't use all of their parameters for every query. When you ask GPT-4 a question, only a fraction of the model's neural network activates. The rest sits idle, waiting for queries that need its particular expertise. This isn't a bug. It's a design choice called Mixture of Experts, and it's the reason modern AI models can be so large without being impossibly expensive to run.
A traditional neural network: what's called a "dense" model: processes every input through every parameter. A 175-billion parameter model uses all 175 billion parameters for every single token it generates. This works, but it's brutally inefficient. Most of those parameters aren't relevant to most queries.
A Mixture of Experts (MoE) model takes a different approach. It splits the network into many smaller "expert" sub-networks. A lightweight "router" network looks at each input and decides which experts to activate. For any given query, only a subset of experts fire: typically 2 out of 8, or 8 out of 64.
The result: a model with 1.8 trillion total parameters might only use 200 billion for any single query. You get the knowledge capacity of an enormous model at the inference cost of a much smaller one.
The MoE architecture has been around since the 1990s, but it was notoriously difficult to train well. Experts would "collapse": the router would learn to send all inputs to the same few experts while others atrophied. Load balancing was a nightmare.
Recent breakthroughs fixed these problems:
Please enable JavaScript to read the full article.