[ArXiv 2026] DynaMoE: Shattering the "One-Size-Fits-All" Constraints of Mixture-of-Experts

TL;DR

The Mixture-of-Experts (MoE) paradigm has been the go-to for scaling LLMs (like Mixtral or GPT-4), but it has long been shackled by two rigid design choices: activating a fixed number of experts (Top-K) and placing an equal number of experts in every layer. DynaMoE introduces dynamic activation and layer-wise capacity scheduling, proving that where you put your experts is just as important as how many you have.

The Problem: The Inefficiency of Rigid Sparsity

Current MoE architectures treat all tokens and all layers as equals. However, in a neural network:

1. Not all tokens are equally hard: A "simple" token in a sentence shouldn't require the same number of active parameters as a complex, ambiguous one.
2. Not all layers do the same work: Early layers often handle raw feature extraction (high entropy), while deeper layers perform high-level semantic integration.

The standard uniform MoE approach results in "wasted" capacity in deep layers for CV and under-provisioned early layers, leading to the Generalist Trap—where experts never truly specialize.

Methodology: Dynamic Choice and Adaptive Depth

1. Dynamic Token-Level Routing

Instead of picking the Top-2 or Top-4 experts, DynaMoE uses a percentile threshold ($au$). If a token has one very strong expert match, it might only activate that one. If it's a complex token with multiple "vague" matches, it can activate many. This allows for Adaptive Complexity.

2. Layer-wise Expert Schedules

The authors tested six ways to distribute experts. The most impactful are:

• Descending: More experts at the start (Good for Vision).
• Ascending: More experts at the end (Good for Language).
• Pyramid: Capacity peaks in the middle.

Figure 1: The Descending schedule architecture concentrates parameters where representational diversity is highest.

The "Why": The Representational Diversity-Convergence (RDC) Principle

This is the core insight of the paper. The authors argue that a network's Representational Diversity ($D_{\ell }$) changes with depth.

• In CV: Entropy is highest at the pixel level (Layer 1). As the network progresses, it compresses data into class labels. Thus, early layers need more experts to handle the messiness of raw data.
• In NLP: Complexity often grows as the model integrates longer context and syntax. Hence, flat or ascending schedules become more viable.

Experimental Showdown: Results that Matter

DynaMoE was tested across image classification (MNIST, CIFAR-10) and Language Modeling (Recycling-the-Web).

• Vision Boost: The Descending schedule delivered a +5.47% accuracy gain on CIFAR-10 over the standard MLP baseline.
• Efficiency: DynaMoE reached 95% of its final accuracy 25% faster than standard MLPs because the dynamic routing reduces gradient variance during training.

Figure 2: Probability heatmaps show the Descending schedule (left) successfully concentrating expert usage in early layers.

Critical Analysis & Conclusion

The Takeaway

DynaMoE proves that Uniform MoE is a suboptimal default. For researchers and developers building the next generation of sparse models, the recommendation is clear:

• For Vision/Hierarchical tasks: Use a Descending schedule.
• For Sequential/LLM tasks: Use an Ascending or Pyramid schedule.

Limitations

While the results are promising, the language modeling experiments were conducted on a relatively small "pilot" scale (1,000 samples). The true test will be scaling DynaMoE to billion-parameter Transformers on datasets like SlimPajama or C4.

DynaMoE provides a much-needed theoretical framework for Adaptive Computation, ensuring that every FLOP is spent where it provides the most "diversity" gain.