[ICLR 2025] From Data Statistics to Feature Geometry: How Correlations Shape Superposition
From Data Statistics to Feature Geometry: How Correlations Shape Superposition
The paper introduces Bag-of-Words Superposition (BOWS), a framework for studying how neural networks represent correlated features in over-complete bases. It demonstrates that when features are correlated, interference can be constructive rather than just noise, allowing models to achieve norm-efficient reconstruction by arranging features according to their co-activation patterns (Linear Superposition).
TL;DR
Why do features in Large Language Models (LLMs) cluster semantically or form perfect circles like "months of the year"? Traditional mechanistic interpretability suggests that features are packed into limited dimensions as "noise" to be filtered. This paper flips the script: in realistic data, features use constructive interference to support each other's reconstruction. By introducing the BOWS (Bag-of-Words Superposition) framework, the authors prove that feature geometry is a direct map of data correlations, shaped by weight decay and compression.
The Missing Piece in Superposition Theory
For years, the industry standard for understanding how neural networks "cram" more features than they have dimensions has been Superposition. However, the classic "Toy Models" (Elhage et al., 2022) assumed features are independent. In that vacuum, the best a model can do is arrange features as far apart as possible (like regular polytopes) and use ReLUs to prune the resulting interference.
The Problem: Real features (like "January" and "February") are rarely independent. If a model treated them as uncorrelated noise, it would waste massive energy filtering out signal that actually tells us something. The gap between "Regular Polytopes" in theory and "Semantic Clusters" in practice remained unexplained—until now.
Methodology: Constructive vs. Harmful Interference
The authors define two distinct regimes of representation:
- Linear Superposition: When features are correlated, the interference between them actually aligns with the signal. Like PCA, the model projects features onto a low-rank subspace where the -th feature is partially reconstructed by the -th feature's activation.
- Non-linear Superposition: The classic regime where interference is unstructured noise requiring a ReLU to suppress false positives.
Introducing BOWS (Bag-of-Words Superposition)
To test this, the authors created BOWS. They trained autoencoders to reconstruct "bags of words" from WikiText. Unlike synthetic toys, BOWS contains the messy, rich covariance of real human language.
Figure 1: BOWS illustrates how 'December' can contribute constructively to the reconstruction of 'Christmas', creating a circular arrangement in latent space.
Key Insight: Interference as a Feature, Not a Bug
The most striking finding is the Beatles Effect. For rare words like "Beatles" or "McCartney," the model often cannot reconstruct the word in isolation (the "one-hot" case). However, when the context is rich with related terms, the score jumps significantly.
In 81% of cases, interference from correlated words actually improved reconstruction. The model isn't just "tolerating" the noise; it’s using it as positive evidence.
Figure 2: Correlated words provide positive pre-activation (constructive interference), while the ReLU and negative bias clean up the residual when the target word is absent.
Visualizing the Geometry
The paper demonstrates that as you tighten the bottleneck (decreasing latent dimension ) or increase weight decay, the model is forced to prioritize Linear Superposition. This naturally causes:
- Semantic Clustering: Related words (sports, chemistry, verbs) huddle together in latent space because they share variance.
- Circular Manifolds: The logical "cycle" of months or days is inherited directly from the data's cyclic covariance.
Figure 3: UMAP projections of BOWS embeddings showing clear semantic clusters (e.g., Science, Sports) emerging from compression.
Presence-Coding vs. Value-Coding
Finally, the authors resolve a potential confusion: Not all circles come from correlations.
- Presence-coding (BOWS): Structure comes from "What words appear together?"
- Value-coding (Modular Addition): Structure comes from "What is the mathematical value?" (e.g., sine/cosine). By ablating subspaces, they prove that LLMs use both, but for very different reasons. One is a map of the world (statistics), and the other is a tool for calculation (logic).
Conclusion: A New Lens for MI
This work suggests that the "Linear Representation Hypothesis" is more robust than previously thought. The complex "manifolds" we see in LLMs might not be a failure of linearity, but rather Linear Superposition at work—efficiently encoding the world's correlations into a low-dimensional bottleneck. For researchers building Sparse Autoencoders (SAEs), this is a wake-up call: your dictionary features are not independent atoms; they are part of a highly structured, constructive geometry.