The Linear Centroids Hypothesis: Beyond Latent Space to Local Experts

TL;DR

For years, the "Linear Representation Hypothesis" (LRH) has been the bedrock of AI interpretability, assuming features are directions in latent activation space. This paper proposes a paradigm shift: the Linear Centroids Hypothesis (LCH). By looking at the centroids of the network's geometric partition—essentially the "DNA" of a model's local experts—we can identify features that are more semantically coherent, less spurious, and directly tied to the model's actual computation.

Problem: The Mirage in Latent Space

Current state-of-the-art interpretability tools, like Sparse Autoencoders (SAEs), rely on latent activations. However, this approach has a "spurious feature" problem. Just because a feature (like "color") is linearly separable in the latent space doesn't mean the network is actually using it to make a decision.

The authors reveal that activations often represent features that are merely "extracted" but not "utilized." Furthermore, linking these latent directions back to specific neurons or layers remains a messy, post-hoc process.

Methodology: High-Dimensional Tiling and Local Experts

Deep Networks (especially ReLU-based ones) can be viewed as complex "tiling" machines. They divide the input space into millions of tiny convex regions (polytopes). In each region, the network behaves like a simple linear "local expert."

The LCH posits that features are not just directions where data points live, but directions where these local experts align.

• The Centroid ($\mu$): The researchers represent each local expert by its centroid.
• The Jacobian Link: Mathematically, the centroid of a region is the row-sum of the network's input-output Jacobian ($J^{o}p1$).
• Mechanism: This allows us to calculate centroids for any differentiable sub-component (a single layer, a block, or the whole model), creating a unified language for interpretability.

Figure 1: The LCH demonstrates that centroids of local experts align into linear directions for semantically consistent features, such as the edges of a star polygon or saliency maps in vision transformers.

Key Results: Cleaner Dictionaries and Better Generalization

The authors tested LCH by swapping activations for centroids in standard interpretability pipelines:

1. Smarter Feature Dictionaries: SAEs trained on centroids (LCH) rather than activations (LRH) produced features with much higher semantic consistency. For example, an LCH "golf ball" feature correctly clustered white golf balls, whereas the LRH version was confused by color and shape.
2. Circuit Discovery: In GPT-2 Large, the authors used a centroid-based attribution metric to find the specific neuron responsible for predicting the word "an." This neuron appeared in the top 99.8th percentile, effectively automating the "needle in a haystack" search for functional circuits.
3. Generalization: Centroid-based probes generalized better across different datasets (e.g., from "truthfulness" to "plausibility") in Llama-3.1-8B, proving that centroids capture the action of the network better than raw activations.

Figure 2: LCH-derived feature dictionaries (blue) exhibit significantly higher accuracy on downstream tasks and generalization to unseen inputs compared to LRH-derived ones (orange).

Deep Insight: Saliency that Makes Sense

One of the most striking applications is a new type of saliency map: the Local Centroid. Traditional gradient saliency maps often look like noise or stay the same even if the network is untrained. The Local Centroid method, which averages centroids in a small neighborhood around an input, produces vividly clear maps that highlight exactly what an "adversarially robust" model sees.

Figure 3: Comparing standard centroids to the "Local Centroid" method. The robustly trained model (bottom right) shows much clearer alignment with the actual object features.

Conclusion & Perspective

The Linear Centroids Hypothesis is more than a technical tweak; it is a shift toward a mechanistic geometry. By focusing on the Jacobian-based centroids, we effectively "listen" to what the network is doing rather than just "looking" at where the data goes.

While this requires slightly more computation (about 10-15% for Jacobian vector products), the payoff is a clearer, more robust, and more hierarchical understanding of how AI models actually think. Future work will likely look into the "radius" of these regions to complete the geometric picture.