All Circuits Lead to Rome: The Fall of the Unique Mechanism Myth in LLMs
All Circuits Lead to Rome: Rethinking Functional Anisotropy in Circuit and Sheaf Discovery for LLMs
The paper introduces Overlap-Aware Sheaf Repulsion (OASR) to challenge the "Functional Anisotropy Hypothesis," which assumes LLM tasks rely on unique internal mechanisms. It demonstrates that a single LLM task is supported by multiple, structurally distinct circuits (sheaves) that are simultaneously faithful, sparse, and complete.
TL;DR
Current AI interpretability research operates on a massive, hidden assumption: that every task an LLM performs has one specific "circuit" or mechanism behind it. This paper, "All Circuits Lead to Rome," systematically demolishes this Functional Anisotropy Hypothesis. By introducing a new discovery method called Overlap-Aware Sheaf Repulsion (OASR), the authors prove that LLMs actually contain a "plethora" of different, almost entirely disjoint subgraphs that can perform the same task with identical accuracy.
The "One Task, One Circuit" Fallacy
Traditional Mechanistic Interpretability treats an LLM like a literal circuit board. If the model completes the sentence "John and Mary went to the store, John gave a drink to..." with "Mary," we assume there is a specific set of attention heads—the "Indirect Object Identification (IOI) Circuit"—responsible for this.
The authors point out two major blind spots in prior SOTA:
- The Search for the "Smallest" Circuit: We reward minimality, assuming that as we prune, we converge on a unique "essential core."
- Deterministic Discovery: Methods like ACDC or EAP are often used once, assuming that the circuit found is the only circuit.
Methodology: Making Sheaves Repel Each Other
To prove that multiple mechanisms coexist, the authors developed Overlap-Aware Sheaf Repulsion (OASR).
The logic is elegant:
- Discover an initial "sheaf" (a standalone faithful subgraph) using gradient-based optimization.
- Run the discovery again, but this time add an Overlap Loss (). This penalty forces the optimizer to find a different set of edges that still solves the task.
Figure 1: While traditional CSD looks for a single path, OASR uncovers multiple distinct pathways that lead to the same functional result.
The Shocking Reality: 4.1% Overlap
In a case study on the famous IOI task, the authors found two sheaves, A and B. Both achieved 100% accuracy. However, their Intersection-over-Union (IoU) was a mere 4.1%.
They weren't just "slightly different"; they were essentially running on different "hardware" within the same model. This isn't just a "backup" (like the Hydra Effect where a head takes over only after an ablation); these mechanisms are simultaneously present and active during normal operation.
Figure 2: Analysis of structural differences. Note how the distribution of incoming edges for MLPs differs substantially, proving these aren't just superficial reshufflings.
The Three-Edge Sheaf and The "Indispensable" Lie
The authors pushed the limits of sparsity, identifying an ultra-minimal three-edge sheaf for IOI. At first, it looked like the "canonical core." But then came the twist: even these three edges weren't indispensable. If the researchers explicitly prohibited those specific edges, the discovery method simply found other sparse circuits that could still do the job.
This led to the proposal of the Distributive Dense Circuit Hypothesis: Linear superposition in high-dimensional spaces allows for "subset-sum collisions." Essentially, there are so many ways to combine different edge contributions to hit the required "logit margin" that many distinct subgraphs can produce the same final token prediction.
Critical Analysis: What This Means for the Field
This paper is a significant "vibe check" for the mechanistic interpretability community.
The Good:
- It reconciles the "sparse circuit" view with the "distributed representation" view.
- It provides a rigorous mathematical proof (The Existence Theorem) for why non-uniqueness is a feature, not a bug, of high-dimensional models.
The Limitation: The study focuses on relatively simple, well-defined benchmarks (IOI, BLiMP, Docstring). Whether this "plethora of mechanisms" stays as disjoint in complex, high-level reasoning tasks (like multi-step math or creative writing) remains an open question.
Conclusion
Mechanistic explanations are not "canonical." When we say we've "found the circuit for X," we've actually just found one valid realization among a vast space of equivalent mechanisms. Future research must stop looking for the "one true core" and start mapping the functional manifolds of how LLMs distribute their intelligence.