[CVPR 2026] Mixture-of-Depths Attention (MoDA): Solving Information Dilution in Deep LLMs

TL;DR

Scaling model depth has long been a double-edged sword: while it increases representational capacity, it often leads to signal degradation as residual connections dilute critical features from shallower layers. Mixture-of-Depths Attention (MoDA) solves this by allowing each layer to "look back" and retrieve specific information from previous layers using a unified attention mechanism. It achieves significant performance gains (+2.11% on downstream tasks) while maintaining 97.3% of FlashAttention-2's hardware efficiency.

The "Depth Compression" Trap

In standard Transformer architectures (ResNet-style), information moves through a single trajectory. Each layer reads from the previous one, processes it, and adds the result back via a residual connection.

The problem? Compression. By the time a signal reaches layer 80, the sharp, informative features from layer 5 have been "diluted" by 75 subsequent residual updates. While DenseNet-style architectures tried to fix this by connecting every layer to every other layer, they failed at scale because the computational cost exploded quadratically.

Methodology: "Read, Operate, Write" through Depth

The authors propose a shift in perspective, viewing the depth stream through three steps:

1. Read: Instead of just reading the identity from the previous layer, MoDA reads a set of historical Depth KV pairs.
2. Operate: Each query head attends to both the sequence (tokens in time) and the depth (the same token across previous layers) simultaneously in one softmax.
3. Write: The layer outputs its results and appends its new KV states to a "Depth Stream" for future layers to access.

Figure: Comparing Depth Residual, Depth Dense, and the proposed Mixture-of-Depths (MoDA).

Why Unified Softmax Matters

Unlike previous methods that might use separate gates or projections, MoDA uses a Unified Softmax. This means the model dynamically decides whether a specific attention head should focus on context (sequence) or historical representation (depth). This competition for "attention probability" ensures that the model only retrieves depth information when it is truly more useful than the sequence context.

Engineering the Efficiency: Hardware-Aware Fusing

A naive implementation of "looking back at all layers" would be a nightmare for GPU memory controllers (HBM). To solve this, the authors introduced several CUDA-level optimizations:

• Flash-Compatible Layout: Flattening the depth cache so each query finds its historical states in a contiguous memory block.
• Chunk-aware & Group-aware Indexing: Exploiting Grouped Query Attention (GQA) to reuse memory fetches, reducing HBM traffic by a factor proportional to the group size $G$.

Figure: Reorganizing memory layout to transform irregular depth lookups into contiguous GPU block reads.

Experimental Validation

The researchers tested MoDA at 700M and 1.5B parameter scales using the OLMo2 recipe.

Key Breakthroughs:

• Downstream SOTA: In the 1.5B setting, MoDA outperformed strong baselines on 10 benchmarks, with standouts in reasoning tasks like ARC-Challenge (+4.35%) and WinoGrande (+2.37%).
• Perplexity Reduction: Consistent improvements across diverse domains (C4, Wiki-text, Pile), proving that the model learns more efficiently when depth is accessible.
• Attention Sink Mitigation: Visualization shows that MoDA redistributes "wasted" attention mass from sink tokens toward useful depth-KV entries, making better use of the model's fixed attention budget.

Table: MoDA consistently outperforms OLMo2 across almost every reasoning and knowledge benchmark.

Critical Analysis & Future Outlook

MoDA represents a significant step toward "Non-Markovian" depth processing in Transformers. However, some challenges remain:

1. Memory Scaling: In extremely deep models (e.g., 100+ layers), caching all historical KV states might lead to memory bottlenecks. The authors suggest "Bounded Depth-KV" as a future remedy.
2. Post-Norm vs Pre-Norm: Interestingly, MoDA performs better with Post-Norm, suggesting that the "retrieval" mechanism interacts differently with the gradient flow than standard residual paths.

Conclusion: By treating the model's own layers as a retrievable memory bank, MoDA effectively breaks the performance ceiling of deep Transformer stacking, paving the way for more efficient and robust trillion-parameter models.