Tapered Language Models: The "Free Lunch" Hidden in Layer Depth
Tapered Language Models
The paper introduces Tapered Language Models (TLMs), an architectural principle that monotonically reduces parameter capacity across the depth of a model under a fixed total budget. By focusing on tapering MLP intermediate width using a smooth cosine schedule, the authors achieve SOTA perplexity and downstream reasoning improvements across Transformer, Gated Attention, and Memory-based architectures.
TL;DR
For years, we have built Large Language Models (LLMs) like identical LEGO bricks stacked on top of each other—every layer has the same width and the same number of parameters. Tapered Language Models (TLMs) challenge this "uniformity myth." By simply making earlier layers wider and later layers narrower (using a smooth cosine schedule), researchers from Mila and Cornell have found a way to boost model performance across the board without adding a single parameter or a single FLOP to the budget.
The Uniformity Myth: Why All Layers Are Not Created Equal
Since the 2017 "Attention Is All You Need" paper, the "uniform chassis" has been the industry standard. Whether it’s Llama, GPT, or Mamba, every layer gets the same slice of the parameter pie.
However, recent interpretability research suggests that LLMs operate like a manufacturing line:
- Early layers do the heavy lifting, transforming raw tokens into complex syntactic and semantic concepts.
- Later layers act as "refiners," making incremental adjustments to the residual stream rather than radical transformations.
The Insight: If later layers are just "polishing," why do they need the same massive MLP (Multi-Layer Perceptron) capacity as the early ones? The authors found that "wider-early" configurations significantly outperform "wider-late" ones under a fixed budget.
Methodology: The Art of the Taper
The authors focus on the MLP width (d_ff) as the primary lever, as MLPs account for the vast majority of parameters in modern LMs. They compared three schedules for reducing this width as we go deeper into the model: Linear, Sigmoid, and Cosine.
The Winning Formula: Cosine Decay
The Cosine Taper emerged as the clear winner. Unlike the Linear taper (too blunt) or the Sigmoid taper (too sudden), the Cosine schedule offers:
- Soft Plateaus at the start and end, allowing the model to stabilize.
- Graceful Mid-stack Transition, providing a smooth gradient of capacity.
The optimal ratio found was 1.5 → 0.5 (meaning the first layer is 1.5x the baseline width, and the last layer is 0.5x), which drastically improved perplexity compared to a uniform 1.0 baseline.
Why It Works: A Mechanistic Look at "Novelty"
To prove why front-loading helps, the researchers measured Layer-wise Novelty. They looked at the cosine similarity between a layer's output and the residual stream entering it.
As shown in the figure above, as depth increases, the MLP output becomes more aligned with the existing residual stream. This means later layers are essentially "recycling" information rather than creating "novel" features. Tapering acknowledges this reality: it shrinks the layers that have stopped innovating and gives that "innovation budget" back to the early layers where it can actually make a difference.
Experiments: Architecture Agnostic Wins
TLMs aren't just for Transformers. The authors tested the tapering principle on four diverse architectures:
- Standard Transformers (Softmax Attention)
- Gated Attention (Sparse/Sinks-free)
- Hope-attention (Self-modifying memory)
- Titans (Neural long-term memory)
Across all scales (up to 1.3B) and architectures, the Tapered versions consistently beat the Uniform ones in commonsense reasoning (ARC, HellaSwag, PIQA) and perplexity (WikiText, LAMBADA).
Critical Analysis & Future Outlook
The Verdict: Tapering is a rare "free lever" in AI. It requires no new hardware, no custom kernels, and no change in training compute—just a different shape for your weight matrices.
Limitations:
- Optimal Tuning: The study used a 440M model to find the 1.5/0.5 ratio. It’s possible that at 70B or 400B scales, the "ideal" taper looks different (perhaps even more aggressive).
- Other Dimensions: While this paper tapered MLPs, future work could taper Attention Head counts or KV cache dimensions.
Conclusion: Tapered Language Models prove that the geometry of a model matters just as much as its size. As we move toward more efficient AI, depth-aware capacity allocation will likely become the new default, replacing the uniform "standard" that has held us back for nearly a decade.