> <former: Breaking the Uniformity Myth in Transformer Scaling
Variable-Width Transformers
The paper introduces the > <former, a novel Transformer architecture that utilizes non-uniform layer widths. By employing an x-shaped capacity allocation (wider early and late layers, narrower middle layers) and a parameter-free residual resizing mechanism, the authors achieve SOTA efficiency for decoder-only models (200M to 3B parameters), outperforming uniform-width baselines in both language modeling loss and downstream tasks.
TL;DR
For years, we have scaled Transformers by simply stacking identical blocks. The > <former (pronounced "hourglass-former") proves this is suboptimal. By narrowing the middle layers and keeping the ends wide, this new architecture achieves better performance than standard models while slashing training FLOPs by 22% and KV cache memory by 15%.
The Problem: The "Flat" Width Assumption
In a standard Transformer, the hidden dimension () is constant. Whether it's the 1st layer or the 40th, we allocate the same amount of math and memory. However, research into "representation collapse" shows that middle layers often become degenerate, essentially performing low-rank operations in a high-dimensional space. We are paying for capacity that the model isn't using.
The authors ask a fundamental question: If a fixed parameter budget is given, should it be spread evenly, or can we find a better "shape"?
Methodology: The X-Shaped Bottleneck
The authors experimented with various shapes (Growing, Narrowing, Diamond, and X-shaped). The winner was the X-shape (> <), which features:
- Wide Early Layers: To extract high-entropy features from the input.
- Narrow Middle Layers: To act as a bottleneck and regularizer.
- Wide Late Layers: To expand capacity for final token prediction.
Fixed-Residual Resizing
A major challenge with variable width is how to maintain the skip-connection (residual stream) when dimensions change. Instead of expensive projection matrices, > <former uses a "Carry-Forward" mechanism.
As shown in the architecture, the model maintains a "global" residual stream. A narrow layer only updates a slice of this stream; the unused dimensions are simply copied upward. This parameter-free approach outperforms padding or learned projections.
Results: Efficiency Without Compromise
The > <former was tested across dense models (200M to 2B) and MoE models (3B). The results were consistent:
- Lower Loss: Consistently outperforms uniform baselines at any given parameter count.
- Faster Inference: Because the average layer width is smaller, the attention FLOPs and KV cache size (crucial for long-context I/O) are reduced by ~10-15%.
- Better Regularization: The bottleneck forces the model to use its MLP dimensions more efficiently.
The scaling curves (Loss vs. FLOPs) show that not only does the > <former have a better starting point, but it also has a steeper scaling exponent, suggesting the gap will only grow as we scale to GPT-4 sized models.
Deep Insight: Mitigating the Compression Valley
Perhaps the most fascinating finding is why this works. In standard models, the "Normalized Matrix Entropy" of the residual stream collapses in the middle layers—essentially, the model gets "bored" and the representations become redundant.
The > <former prevents this. By physically restricting the width in the middle, it forces the network to maintain a high-entropy manifold. It's the "less is more" principle applied to neural architecture.
Conclusion & Future Outlook
The > <former challenges the "uniformity" dogma of the Transformer era. While current hardware (and kernels) are optimized for uniform blocks, the theoretical gains in KV cache and FLOPs are too significant to ignore. As we push toward "Infinite Context" windows, non-uniform architectures like > <former may become the standard for resource-optimal scaling.
Key Takeaway: Don't scale deeper and wider—scale smarter by shaping the capacity to match the computational needs of the task.