Mixture of experts

A mixture-of-experts model replaces a single dense feed-forward block with many parallel "expert" sub-networks and a learned router that picks a handful of them for each token. The total parameter count can be enormous, but most weights sit idle on any given pass — so the compute cost matches a model far smaller than the full parameter tally suggests. The hard part is the router: if it always picks the same experts, the others atrophy and the sparsity buys nothing. Balancing load across experts during training is what makes or breaks the approach.