[ICLR 2025] Why LLMs Learn "Useless" Features: A Developmental Look at Next-Token Prediction

TL;DR

Why does a model trained solely to predict the very next word bother to learn the entire state of an Othello board or complex syntactic structures? This paper provides a rigorous answer: Next-Token Prediction (NTP) is not a single signal. By decomposing the gradient into Direct, Pre-cached, and Shared components, the authors show that "NTP-useless" features are actually the product of the model preparing information for future consumption (pre-caching) or transferring knowledge across positions (circuit sharing).

The "Usability" Paradox

In a strictly myopic world, a Transformer should only care about what helps it guess the next character. Yet, we see "World Models" emerging—abstract representations of reality that seem overkill for the task.

The authors point out a massive gap in current AI research: we spend all our time looking at what a finished model does (the teleological view) and almost no time looking at the gradient pressure that forced those features to grow in the first place (the developmental view).

The Three Pillars of Feature Growth

The core contribution is a mathematical decomposition of how information flows from the loss back to the parameters $heta$:

1. Direct Learning: The classic view. Feature $X$ at position $i$ helps predict token $i+1$.
2. Pre-caching: Position $i$ computes a feature that isn't useful for $i+1$, but is "picked up" by attention at position $j>i$. The model is essentially calculating "breadcrumbs" for its future self.
3. Circuit Sharing: Because parameters are tied across positions, a feature that is useful at position $j$ might "leak" into position $i$ simply because the same weights are being updated.

Proving it in the "Wild" (Othello & Gemma)

The authors applied this framework to two fascinating cases:

1. The Fragility of OthelloGPT

Previous studies found OthelloGPT's world model is "fragile." This paper explains why:

• NTP-Useful squares (those that change the set of legal moves) receive a strong Direct gradient.
• NTP-Useless squares (empty or irrelevant to the next move) only receive Pre-cached or Shared signals.
• Result: The model learns the useless squares less robustly because the training signal is literally weaker and more indirect.

2. Code and Formal Logic in Gemma 2

By analyzing Sparse Autoencoder (SAE) features in Gemma 2, the authors found a striking pattern. Using a metric $Q(w)$ (the ratio of future vs. immediate influence), they discovered that Pre-cached features are heavily biased toward code and math.

This suggests that formal reasoning requires the model to hold onto structural state (like tracking an AST or nested parentheses) that isn't immediately "outputtable" but is vital for the logic three tokens down the line.

Debunking the "Look-Ahead" Myth

There is a popular theory that LLMs "plan ahead" by predicting future tokens internally. However, this paper found a negative correlation between pre-cached features and a model's ability to predict future tokens (look-ahead).

This supports the "Breadcrumbs Hypothesis": models don't plan the future; they just leave behind clues that are useful for future layers. Look-ahead is likely an accidental byproduct of language's statistical redundancy, not a deliberate model strategy.

Critical Insight & Conclusion

This paper is a wakeup call for mechanistic interpretability. We cannot understand why a model behaves the way it does without looking at the training dynamics.

Takeaway: If you want to build models that reason better, you cannot just look at the final weights. You must understand the "Pre-caching" pressure. The emergence of reasoning is essentially the model learning to talk to itself across time steps.

Limitations: The attribution method requires retraining from scratch, which is impossible for GPT-4 scale models. However, the $Q(w)$ metric offers a clever proxy for existing checkpoints.