[ArXiv 2025] Sparsity vs. The Curse of Depth: How LLMs Scale Effectively

TL;DR

As Large Language Models (LLMs) grow deeper, they often become less efficient, a phenomenon known as the Curse of Depth (CoD). This paper reveals that sparsity—commonly used for speed—is actually a secret weapon for signal stability. By regulating variance propagation, both implicit and explicit sparsity prevent deep layers from becoming redundant "identity mappings," enabling significant gains (up to 4.6% accuracy) when scaling model depth.

The Problem: The Identity Drift

In standard Pre-Layer Normalization (Pre-LN) Transformers, output variance grows sub-exponentially with depth. As this variance accumulates, the residual stream becomes so "loud" that the unique contribution of any single deep layer becomes a "whisper" in comparison.

Mathematically, the Jacobian of deep blocks approaches the identity matrix ($JoI$). Effectively, the model stops learning new features in its back half, wasting billions of parameters on redundant computations.

The Solution: Sparsity as a Variance Damper

The authors propose a fascinating reframing: Sparsity is a variance regulator. By reducing the density of connections or activations, we reduce the energy passed to each layer update, slowing down the variance explosion.

The study explores two flavors of sparsity:

1. Implicit Sparsity: Forced by training conditions like Weight Decay (pushing parameters to zero) and Long Sequence Lengths (peaked attention distributions).
2. Explicit Sparsity: Architectural choices like Grouped-Query Attention (GQA) and Mixture-of-Experts (MoE).

Theoretical Punchline

The paper proves that the variance bound scales with a density factor $\rho$. A smaller $\rho$ (more sparsity) leads to a smaller per-layer variance gain. Essentially, sparse models "stay quiet" longer, allowing deep layers to remain functionally relevant.

Methodology & Architecture

The researchers used three novel metrics to quantify "Layer Effectiveness":

• Causal Score: Impact of skipping a layer on future representations.
• Permutation Score: Sensitivity of the model to swapping layer orders.
• Usefulness Score: The degree of non-linearity a layer provides beyond a simple identity/linear mapping.

Figure 1: Performance gains when scaling depth from 16 to 32 layers using various sparsity modules.

Key Results: Deep Layers Wake Up

The experiments provide a clear causal chain: High Sparsity $o$ Low Variance $o$ High Layer Usefulness.

• MoE Advantage: MoE models exhibited 3x to 6x lower variance than dense models of the same active parameter count.
• GQA Influence: Switching from MHA to MQA (Multi-Query Attention) decreased variance monotonically and increased the "Usefulness Score" by 7%.
• Weight Decay: Strong regularization (within an optimal range) induced parameter sparsity that recovered performance in deep models that were otherwise collapsing.

Figure 2: Deeper models (L=32) show massive variance growth and a corresponding drop in Usefulness compared to shallower models (L=12).

Practical Recipe for Depth-Effective LLMs

The authors conclude with a "rule-of-thumb" for training deep models:

• Context Matters: Train with longer sequences (e.g., $T=4096$) to induce implicit attention sparsity.
• Regularize for Sparsity: Use optimal weight decay ($\lambda \approx 0.1-0.3$) to keep parameter variance in check.
• Go Sparse by Design: Use MoE and GQA/MQA to decouple model capacity from variance accumulation.

Critical Insight & Conclusion

This work fundamentally changes our perspective on sparsity. We no longer just view MoE or MQA as "cheaper" alternatives to dense models; we should see them as structurally superior for deep scaling. By controlling the physics of signal propagation, sparsity allows LLMs to actually use the depth we give them.

Future Outlook: Could we design "dynamic sparsity" that specifically targets layers once their variance crosses a certain threshold? This paper opens the door to a more controlled, mathematically-grounded approach to LLM scaling laws.