The Knowing–Using Gap: Why Your Fine-Tuned LLM Remembers but Can’t Think

TL;DR

Even when an LLM is fine-tuned to "know" a new fact perfectly, it often fails to "use" that fact in multi-hop reasoning. This paper identifies this as the Knowing–Using Gap. Through a novel diagnostic called Self-Patching, the authors prove the knowledge is actually there—it’s just stuck in the wrong layers. By manually relocating these internal signals, they can "force" the model to generalize.

The Mystery of the "Second-Hop" Failure

Imagine teaching an LLM a new fact: "Sydney is located in [Entity X]". The model quickly learns to answer this direct question with 99% accuracy. However, when asked a downstream reasoning question—"What is the capital of the country where Sydney is located?"—the model fails miserably.

This isn't a problem of memory; it's a problem of alignment. The model has the raw data in its weights, but its internal "reasoning circuits" aren't picking up the phone.

(b) Conceptual illustration: LLMs memorize atomic facts but fail to use them in chain reasoning.

Diagnosing the "Stranded" Knowledge: Self-Patching

To figure out why this happens, the researchers developed Self-Patching. They took the internal hidden state (the "representation") of a fact from one layer and manually "patched" it into a different layer during the reasoning task.

Unlike previous methods like Causal Tracing, which requires the model to already know the answer, Self-Patching works on failed cases. It asks: "If I move the information from Layer 5 to Layer 15, does the model suddenly 'get' the answer?"

The Knowledge–Circuit Misalignment Hypothesis

The study revealed a fascinating spatial dynamic:

1. Early/Late Storage: New knowledge tends to get stored in very early or very late layers.
2. The Mid-Layer Bottleneck: Multi-hop reasoning happens primarily in the middle layers.
3. The Disconnect: Fine-tuning often achieves memorization by fitting the facts into "easy-to-access" storage layers, but it fails to build the "routing" necessary to send that info to the mid-layer processing units.

Figure: Knowledge "permeation" during training. In successful cases (top), the knowledge reaches the diagonal (natural use). In failures (bottom), it halts before the diagonal.

Engineering a Solution: The 75% Heuristic

The most exciting part of this research is that you don't need a complex "Oracle" to fix this. By analyzing where the most effective patches occurred, the authors found a consistent pattern:

• Cluster 1: Information in early layers (~10% depth) needs to move to middle layers (~50% depth).
• Cluster 2: Information in late layers (~80% depth) needs to move back to middle layers.

Using a fixed heuristic (pre-determined layer pairs), they recovered 58–75% of the performance gap across different models like LLaMA-3 and Qwen-2.5.

Effective patch locations concentrate into two primary clusters (Early-to-Mid and Late-to-Mid).

Why This Matters for the Industry

As we move toward "continual learning" and real-time knowledge updates for LLMs, simply "fitting" new data isn't enough. This paper proves that:

• Memorization $eq$ Generalization: We need better evaluation metrics than just training loss.
• Routing is the Key: Future fine-tuning techniques (like "Alignment-Aware Training") should focus on ensuring new information is integrated into the model's existing reasoning pathways, not just stored in a corner of its weights.

Conclusion

The Knowing–Using Gap isn't a sign that LLMs are "too small" to learn new things; it’s a sign that our current fine-tuning methods are inefficient at internal routing. By understanding the mechanics of how knowledge "permeates" through layers, we can start building models that don't just echo facts, but actually understand how to use them.