Attractor Geometry: Why Your LLM Hallucinates with Such Confidence

TL;DR

LLMs are often "confidently wrong." This paper reveals why: the internal hidden states of a Transformer know when they are lost, but the frozen output head (LM Head) erases this doubt. By viewing memory as a landscape of "attractor basins," researchers from MIT and Cognizant show that we can detect hallucinations with near 100% accuracy by measuring how far a hidden state is from a "memorized basin," a metric they call Geometric Margin.

The "Confidence" Bottleneck

Why does a model hallucinate? Conventionally, we look at Output Entropy (the randomness of the word choices). However, as models get bigger, they become "overconfident." A 70B model might predict a total lie with the same 99% probability it uses for a known fact.

The authors argue that the LM Head—the final layer that turns numbers into words—is an epistemic bottleneck. It is trained to pick a winner, not to report on the model's internal confusion.

Methodology: The Geometry of Memory

The core insight is that Parametric Memory (PM) is not just a lookup table; it’s a dynamical attractor.

• MLP Layers: These carve out "basins" in the representation space. If a query is "known," the hidden state is sucked into a specific basin.
• QK (Attention) Layers: These act as "routers," steering the state toward the right basin.
• Working Memory (WM): Contextual information provides a "transient pull" that competes with these persistent basins.

Jacobian Decomposition

To prove these roles aren't just fine-tuning artifacts, the authors used Jacobian Decomposition on pretrained models. They found that the MLP layers have the highest contractive magnitude (forming basins), while the VO (Value-Output) layers exhibit "Hopfield-like" symmetric properties for readout.

Figure 1: The dissociation of memory roles: QK routes, MLP carves the landscape, and VO performs the readout.

Two Modes of Failure

1. Conflict (Context Deafness): When your prompt says "The code is 123" but the model has memorized "456," the two memories fight. If the memorized "basin" is too deep, the model ignores your context entirely.
2. Hallucination (Basin Absence): When the model is asked about something it never learned, no basin exists. The hidden state wanders aimlessly, yet the LM head still forces it to choose a token confidently.

The Geometric Cure: Margin vs. Entropy

The researchers proposed a new diagnostic: Geometric Margin ($\delta$). This measures the Euclidean distance from the final hidden state to the nearest "known" basin center.

• Success: Margin is low.
• Hallucination: Margin is high, even if output entropy is near zero.

Figure 2: Geometric Margin vs. Output Entropy. Notice how margin (blue/red separation) provides a much cleaner signal than entropy for detecting hallucinations.

In tests on natural language facts (like "What is the capital of Palau?"), the margin achieved an AUROC of 1.000. It perfectly separated correct answers from hallucinations. Meanwhile, output entropy was significantly less reliable (AUROC 0.622).

The Scaling Law of Hallucination

Perhaps the most startling find is that hallucinations get harder to detect as models grow. The authors derived a scaling law: $C=\exp (-c/\bar{\Delta })$. As models scale, the "logit gap" ($\Delta$) increases due to softmax saturation. This means the model's output looks increasingly certain even when it is making things up. Paradoxically, while larger models hallucinate less often, their hallucinations are more confident and harder to catch with standard tools.

Critical Insight & Future Work

The internal world of a Transformer is rich with "epistemic" information—it knows what it doesn't know. However, our current way of "reading" models via the LM head throws this information away.

The Takeaway: To build safer AI, we need to stop looking at the "words" the model says and start looking at the "geometry" of where its thoughts are landing. Future architectures might include "Geometric Readout Heads" that bypass the LM bottleneck to flag when a model is wandering outside its known basins.

Limitations

While powerful for factual recall, this method struggles with adversarial reasoning (like TruthfulQA), where a model might have a very deep, well-formed basin for a "common misconception." In those cases, the model is "confidently wrong" because it thinks it knows the answer, not because it is lost.