[arXiv 2024] Self-Routing: Is the Learned Router in MoE Redundant?

Executive Summary

The Mixture-of-Experts (MoE) architecture has become the backbone of modern LLM scaling (seen in Mixtral, DeepSeek, etc.). Traditionally, we assume that choosing which expert should process a token requires a specialized "brain"—a learned router module. Self-Routing challenges this dogma by demonstrating that a simple slice of the hidden state can serve as the router, removing parameters entirely while actually improving load balancing. It positions itself as a structural simplification that proves sparse models can self-organize without explicit gating supervision.

Motivation: The Hidden Cost of "Smart" Routing

In current SOTA MoE models, routing is typically:

1. Parameterized: Requires an $HimesN$ weight matrix per layer.
2. Unbalanced: Often collapses into using only a few experts, requiring a carefully tuned "Load Balancing Loss."
3. Complex: Adds overhead in implementation and synchronization.

The authors' core Insight is: If a Transformer is powerful enough to process language, its hidden state $(h)$ must already contain the features needed for routing. Why project $h$ into a new space when we can just read the routing preference directly from a designated corner of $h$?

Methodology: The "Slice" Technique

Self-Routing is elegantly simple. Instead of calculating $z=hW$, it defines: $z_{self}=h_{[H-N:H]}$ Where $N$ is the number of experts. The last $N$ coordinates of the hidden vector are treated as the "votes" for each expert.

Figure 1: Comparison between the standard Learned Gate (left) and the parameter-free Self-Routing Gate (right).

By forcing the model to use specific dimensions for routing, the backbone is incentivized during backpropagation to store "routing-relevant" features in those specific slots. This is a form of implicit specialization.

Experimental Performance & Expert Utilization

The authors tested this on GPT-2 (language) and DeiT (vision). The results show that not only does performance hold up, but the "quality" of routing improves.

Performance Comparison (GPT-2 Scale)

The "Dead Expert" Solution

One of the most striking findings is the Routing Entropy. Learned routers often become "lazy," sending all tokens to one or two "generalist" experts. Self-Routing, by contrast, naturally distributes tokens more broadly.

Figure 2: Layer-wise entropy. Self-Routing (blue) consistently achieves higher entropy (closer to 1.0), indicating more uniform and efficient use of the expert pool.

Critical Analysis & Conclusion

Why does it work?

Self-Routing suggests that routing is more about representational organization than it is about the "logic" of the router. By removing the learned projection, we remove a layer of abstraction that might otherwise oscillate or collapse during training.

Limitations

• Scale: This was tested at GPT-2 scales (~500M params). Whether this holds for 100B+ parameter models where routing dynamics are more volatile remains an open question.
• Fixed Subspace: Using the same coordinates across all layers might be restrictive. Future work could explore "coordinate hopping" or dynamic subspaces.

Takeaway

Self-Routing is a "less is more" breakthrough. It simplifies the MoE block, removes the need for auxiliary losses, and proves that the internal manifold of a Transformer is rich enough to direct its own data flow. For researchers building next-gen sparse models, this suggests that the route to efficiency might be internal reorganization rather than external complexity.