[CVPR 2026] Mixture-of-Depths Attention: Bridging the Gap Between Depth and Efficiency in LLMs

Executive Summary

TL;DR: MoDA (Mixture-of-Depths Attention) is a breakthrough architectural primitive that allows Large Language Models (LLMs) to "look back" at previous layers' states as easily as they look at previous tokens. By unifying sequence-level and depth-level attention into a single, hardware-optimized operator, it achieves SOTA performance (2.11% gain on downstream tasks) while maintaining 97.3% of the efficiency of FlashAttention-2.

In the current context of LLM scaling, depth has often been the "under-exploited" dimension compared to width and data. MoDA changes this by providing a mathematically sound and computationally cheap way to scale depth without the usual signal degradation.

Problem & Motivation: The "Information Dilution" Trap

As we stack Transformer layers, we rely on the Residual Stream to carry information. However, each layer performs an additive update ($x_{l+1}=x_l+f(x_l)$). Over 80+ layers, the unique features learned in Layer 10 might be "diluted" or buried under the noise of subsequent updates.

Existing solutions like DenseNet (connecting every layer to every other layer) solve this but at a catastrophic cost in parameters ($O(L^2{D}^2)$). The authors of MoDA recognized that Attention itself is the perfect tool for this: why not let the model choose which previous layers' features are still relevant?

Methodology: Mixture-of-Depths (MoDA)

MoDA's core innovation is the Unified Depth-Sequence Search. In a standard Transformer, a Query ($Q$) looks at Keys ($K$) across the time dimension. In MoDA, the Query looks at:

1. Sequence KV: Keys/Values of other tokens in the current layer.
2. Depth KV: Keys/Values of the same token from previous layers.

These are concatenated, and a single Softmax is applied across both. This allows the model to dynamically balance between processing temporal context (sequence) and refined internal representations (depth).

The Hardware Secret: Specialized Triton Kernels

Standard PyTorch implementations of cross-layer indexing are notoriously slow due to non-contiguous memory access. MoDA introduces:

• Flash-Compatible Layout: Flattening the depth cache for contiguous reads.
• Group-Aware Indexing: Aligning with GQA (Grouped Query Attention) so that multiple query heads can share depth-lookup overhead.

Figure 1: Evolution from Residual to Dense to MoDA. MoDA (d) provides the adaptive retrieval of DenseNet with the efficiency of Residual connections.

Experiments & Results: Real Gains at Scale

The authors tested MoDA on 700M and 1.5B models using the rigorous OLMo2 training recipe (400B tokens).

1. Performance vs. Overhead

MoDA achieves significant wins on benchmarks like WinoGrande (+2.37%) and ARC-Challenge (+4.35%). Crucially, the FLOPs increase is a negligible 3.7%.

2. Efficiency Benchmarks

A major highlight is the kernel's performance. As sequence length increases (64K), the "Extra Time" cost of MoDA drops to just 2.73%. This makes it one of the few "complex" attention variants actually viable for production.

Table 1: 1.5B Model performance showing consistent improvements over the OLMo2 baseline.

Depth Insights: Why does it work?

Attention visualizations reveal that MoDA blocks "active retrieval" throughout the depth of the model. Interestingly, MoDA seems to mitigate the "Attention Sink" problem (where models dump probability mass on the first token). Instead, it redistributes that mass to useful depth-wise features, suggesting that "sinks" might actually be a symptom of a model having nowhere else to put its attention mass when current-layer features are insufficient.

Figure 2: Heatmaps showing substantial attention mass being assigned to depth-KV blocks (right of the red dashed line).

Critical Analysis & Future Work

Takeaway: MoDA is a rare example of a "dense-like" connection that is actually practical. It effectively treats the "depth" of a model as a high-speed memory cache.

Limitations:

• Memory Growth: While FLOPs are low, storing KV caches for all layers at all times increases VRAM usage.
• KV Slot Caching: The authors suggest future work involving "Bounded Depth-KV" (keeping only the top-$S$ most useful layer states) to solve the memory scaling bottleneck.

Future Outlook: MoDA could become the default primitive for "Extreme Depth" models (1000+ layers), where standard residual streams inevitably fail due to signal decay. Its architecture-agnostic nature also makes it a prime candidate for Vision and Multimodal models.