The Myth of Expert Specialization in MoEs: Why Routing Reflects Geometry, Not Domain Expertise

TL;DR

Is the "math expert" in an MoE model actually specialized in math? A new study suggests the answer is more complex—and perhaps more mechanical—than we thought. By analyzing models like DeepSeek and Qwen, researchers found that expert selection is a direct consequence of hidden state geometry. Routing isn't about semantic understanding; it's a linear mapping where tokens with similar representations are mathematically forced to pick similar experts.

Problem & Motivation: The Illusion of Knowledge

In the world of Mixture-of-Experts (MoE), we often talk about "expert specialization" as if the model is a panel of specialists. We see "Expert 7" activate for coding and "Expert 12" for law. However, this creates a false sense of interpretability.

The authors argue that existing research focuses too much on which experts activate (Cataloging) rather than why they activate (Mechanism). If specialization were truly semantic, two different models solving the same calculus problem should use similar experts. They don't. The truth lies deeper in the mathematics of the representation space.

Methodology: Routing as Data Geometry

The core insight of this paper is that the MoE router is just a linear projection. If the hidden states $h_i$ and $h_j$ are close together in the representation space, their routing logits $P{h}_i$ and $P{h}_j$ must be close.

The Geometry-Aware Bound

The authors propose a "Data-aware Lipschitz bound" (Proposition 1) that shows how the router alignment with the data's principal subspace determines expert usage.

Figure: The "Triangle" of Routing. When hidden state distance is small, logit distance must be small. As hidden states diverge, the routing can vary wildly.

The Role of Load Balancing

Why don't all tokens just use the most popular expert? The authors prove that the load-balancing loss used during training forces the router to "ignore" the shared directions (the most common features) of tokens. This forces diversity, but it also means that if a batch isn't diverse enough, the model can lose its specialization entirely—a phenomenon called "specialization collapse."

Experiments: Debunking Human Intuition

The paper presents three striking experiments that challenge our understanding of MoEs:

1. Model Divergence on Identical Content

Researchers tested 60 different models on the same set of math questions. If specialization were semantic, the "Math Expert" coverage would overlap. Instead, the expert overlap between two different models solving the same question was only ~60%—no better than the overlap between a single model answering two completely different questions.

2. The Prefill vs. Generation Gap

One of the most surprising findings is that the Prompt does not predict the Rollout. During the prefilling stage (reading the prompt), many models exhibit "Router Collapse" where different prompts activate the same experts. Only when the model starts generating new tokens (rollout) do the expert paths diverge.

Figure: Expert activation frequencies for unrelated topics (HLE questions vs. Wimbledon history). In later layers, they engage an identical set of experts during prefilling, only diverging during actual generation.

3. Contextual Dependency

The same token (e.g., "apple") will route to different experts depending on whether it appears in a "daily fruit" context or a "coding/tech" context. This effect becomes much stronger in deeper layers, where self-attention has allowed the context to "infect" the individual token's hidden state.

Deep Insight & Conclusion

The study concludes that understanding MoE specialization is exactly as hard as understanding the geometry of LLM hidden states—a notorious "black box" in AI research.

Key Takeaways for Practitioners:

• Efficiency vs. Interpretability: MoEs are efficient because they are sparse, not because they are "smart" about dividing labor semantically.
• The Depth Factor: Early layers are token-dependent; deep layers are context-dependent and output-predictive.
• Debugging MoEs: If your model isn't specializing, it might be a data diversity or load-balancing issue, not an architectural failure.

In summary, the "experts" in MoEs aren't specialists in the human sense; they are simply bins that partitioned the high-dimensional geometry of the model's internal thoughts.