The Spike, the Sparse, and the Sink: Decoupling the Hidden Mechanics of LLMs

TL;DR

Researchers from NYU and Meta (including Yann LeCun) have finally performed a "medical autopsy" on two of the most mysterious behaviors in Modern Transformers: Massive Activations (huge numerical outliers) and Attention Sinks (tokens that "suck up" all the attention). The verdict? They aren't mysterious "emergent intelligence"—they are architectural artifacts of the Pre-Norm design. By changing how we normalize or gate attention, we can kill the spikes without hurting the model.

The Mystery: Why are LLMs so "Spiky"?

If you look at the internal tensors of a model like Llama-3 or Qwen, you'll see something strange. A few hidden channels have values thousands of times larger than the rest. Strangely, these "spikes" always happen on the same tokens—usually the very first token or a period—and these same tokens attract almost all the attention from the model's heads.

Until now, we didn't know if the model needed these spikes to think, or if they were just a glitch. This paper proves they are decoupled:

1. Massive Activations act as global implicit parameters.
2. Attention Sinks act as local modulators to help heads "ignore" irrelevant context.

Methodology: The Life Cycle of a Spike

The authors discovered that spikes don't just grow gradually; they have a dedicated life cycle within the layers of the model:

• Step-up Blocks: Early layers (e.g., Layer 4 in Llama 2) use the SwiGLU feed-forward block as a Directional Quadratic Amplifier. If a token aligns with a certain "trigger direction," its value is squared and boosted into the thousands.
• Residual Accumulation: Because of the additive nature of residual connections, these massive values persist across the "plateau" of middle layers.
• Step-down Blocks: Near the very end (the last 1 or 2 layers), the model systematically injects an additive inverse to "neutralize" the spike before the final prediction.

Figure 1: The "Rise-Plateau-Fall" trajectory of activation magnitudes across depths.

How Spikes become Sinks

The bridge between a numerical spike and an attention sink is Normalization (RMSNorm). When a vector with a massive outlier is normalized, two things happen:

1. Sparsification: The non-spike channels are crushed to nearly zero.
2. Constant Vector: Different "spike tokens" (like a newline vs. the first token) all collapse into the same near-constant sparse vector.

This "constant" vector provides a perfect Geometric Anchor. Attention heads can easily learn to point their "Key" projections toward this anchor, creating a stable "dumping ground" for attention mass when the head has nothing useful to look at.

Figure 2: t-SNE visualization showing how Sink Keys (k_s) are geometrically separated from normal keys (k_n).

Experimental Proof: Break the Spike, Keep the Performance

The most impactful part of this study is the Ablation Study. By testing different architectures, the authors proved that these phenomena are not "functional necessities."

• QKNorm & Sandwich Norm: By applying normalization to the Query/Key or at the block output, they eliminated the spikes (reducing magnitudes from 3800 to 92). The model's Perplexity didn't change.
• Conditional Gating: By adding a dynamic gate to the attention mechanism, the Attention Sinks disappeared. This proves sinks are just a "learned workaround" for models that lack explicit routing.

Table 1: Effects of different optimization and architectural choices on Spikes and Sinks.

Critical Insight: Why Does This Matter?

This paper is a major win for practitioners in Model Compression and Long-Context Inference:

1. Quantization: We now know that "outlier channels" (which ruin 8-bit or 4-bit quantization) can be prevented at training time using Sandwich Norm or QKNorm without losing accuracy.
2. KV-Cache Management: Since we know "Sinks" are just a dumping ground for local attention, we can more safely design streaming LLMs that only keep the first token and a sliding window of recent tokens.
3. Future Architecture: The discovery that sinks disappear with conditional gating suggests that the next generation of models (like hybrid Attention-SSM models) might naturally be free of these artifacts.

Conclusion

The co-occurrence of massive activations and sinks is an "incidental architectural interaction." We built Transformers with Pre-Norm and Softmax; the models merely exploited these mathematical loopholes to create stability. As we move toward more robust normalization and gating, the "Sinks" and "Spikes" may soon become relics of the past.