[2026] Transformers Converge to Invariant Algorithmic Cores: Finding the Ghost in the Machine

TL;DR

Why do two neural networks, trained on the same data with different seeds, look like total strangers internally despite reaching the same accuracy? This paper argues that while their weights diverge, their algorithmic cores—compact, causal subspaces that hold the "computational essence"—are identical. By extracting these cores, the author recovers the ground-truth logic of Markov chains, explains why "grokking" happens, and finds a single "grammar switch" in GPT-2 that controls singular/plural generation across all model scales.

The Problem: The "Illusion" of Circuitry

In the field of Mechanistic Interpretability, researchers love "circuits"—mapping specific attention heads to specific tasks. But there is a trap: Underdetermination. Training constrains behavior, not wiring. If you train ten models, you might get ten different circuits.

The author, Joshua S. Schiffman, suggests we stop asking "How is this specific model wired?" and start asking "What functional structure is preserved across all models that solve this task?" This is the shift from Implementation to Invariants.

Methodology: Algorithmic Core Extraction (ACE)

Schiffman draws from Control Theory (specifically Kalman decomposition) to find the "Minimal Realization" of a task. The ACE method looks for a subspace that satisfies two conditions:

1. Active: The model's hidden states actually move in this direction (high variance).
2. Relevant: Moving in this direction significantly changes the output (high Jacobian sensitivity).

By performing SVD on the interaction between activations and sensitivity, we get a Core. The author validates these cores using Causal Ablations:

• Necessity: If you remove the core, the model's performance collapses to chance.
• Sufficiency: If you only keep the core and delete everything else, the model stays at SOTA performance.

Key Insight 1: Grokking and the "Over-Education" Paradox

One of the paper’s most fascinating experiments involves Modular Addition (a task used to study "grokking").

• Grokking as Crystallization: Before the model "gets it," the core is messy. At the moment of grokking, the core "snaps" into a cyclic, rotational structure.
• The Weight Decay Paradox: Usually, we think Weight Decay (regularization) makes models simpler. But Schiffman found it actually makes cores larger (Causal Inflation).

The Reason? Minimum-norm redundancy. Under regularization, the model realizes it's "cheaper" (in terms of weight norm) to distribute the computation across many redundant modes rather than concentrating it in one. This leads to a beautiful inverse scaling law for grokking time: $a{u}_{extgrok}\propto 1/(\omega p)$.

Key Insight 2: The Universal Grammar Axis in GPT-2

Scaling up to GPT-2 (Small, Medium, and Large), the author looked for Subject-Verb Agreement.

Remarkably, in all three models, a 1D Core (a single vector axis) emerged in the final layers. This axis functions as a "Grammatically Correctness" coordinate.

• Linear Control: Projecting hidden states onto this axis predicts the singular-vs-plural preference with near-perfect correlation across all scales.
• Steering: By "flipping" this 1D axis, the author could force GPT-2 to generate sentences with perfect but inverted grammar (e.g., "The key to the cabinets are..."). This worked even for words the core wasn't trained on, proving it captured a global "grammatical number" variable.

Why This Matters: From "Wiring" to "World Models"

The paper concludes that Transformers aren't just memorizing heuristics; they are recovering the generative process of their environment.

• When trained on Markov chains, the core recovers the transition matrix.
• When trained on modular math, the core discovers group symmetry.
• When trained on language, the core recovers discrete linguistic variables.

Limitations: Can this scale to 400B parameter models? While the 1D nature of the GPT-2 cores suggests dimensionality might not increase with scale, the complexity of target tasks (like "reasoning") might make core-extraction much harder.

Conclusion

Schiffman’s work provides a powerful new lens for AI safety and interpretability. If we can find the "Cores" for safety-relevant behaviors (like deception or power-seeking), we could potentially build "switches" to turn them off, regardless of how the weight-level wiring changes between model versions. Target the essence, not the implementation.