Stored but Unreachable: Why Your LLM "Forgets" the Facts You Taught It
Can a Language Model Learn Facts Continually in Its Weights?
This paper investigates whether Large Language Models (LLMs) can effectively learn and retain facts within their weights through sequential fine-tuning. Using Qwen3-4B models and invented facts, the author demonstrates that broad training data is essential for creating usable knowledge, but ultimately finds that "weight writes" fail as a reliable long-term storage medium due to lost reachability rather than erased storage.
Executive Summary
In the quest to move beyond static training sets, the AI community has chased the dream of Continual Learning: the ability to write new facts directly into a model's weights like a hard drive. However, a provocative new study using the Qwen3 architecture suggests we might be looking at the problem fundamentally wrong.
The paper, "Can a Language Model Learn Facts Continually in Its Weights?", reveals that while we can successfully bake "usable" knowledge into weights using diverse training data, these facts don't actually disappear when the model "forgets" them. Instead, the address to that information is stolen by newer facts. The weights store the content, but they fail to provide a stable "address" for retrieval.
The "Recitation-to-Use" Gap: Why Bare Facts Fail
A common pitfall in fine-tuning is teaching a model to parrot a sentence (Recitation) without teaching it the underlying logic (Use). The author identifies a critical metric called the Entailment Gap.
- Bare-statement training: Results in high "Recall" (the model can repeat the fact) but fails miserably at "Counterfactual" or "Composition" tasks (using the fact in a new context).
- Study training: By providing 24 diverse restatements and implications, the model internalizes the logic. This narrowed the entailment gap from 27.4 points to a mere 5.4.
Figure 1: Comparison between writing to weights (which erodes over time) and keeping facts in context (which remains stable).
Methodology: The Anatomy of a Write
The study followed a rigorous protocol:
- Creation: Writing invented facts into a Qwen3-4B model.
- Accumulation: Performing 20 to 100 subsequent writes of unrelated facts.
- Probing: Testing five types of questions: Recall, Paraphrase, Application, Composition, and Counterfactual.
The author found that the breadth of training data is the primary predictor of survival. If a fact was learned via broad study data, it had a 46% chance of surviving 20 sequential writes. If it was a "bare statement," it essentially vanished (1% retention).
The "Ghost" in the Weights: Forgetting is an Access Problem
The most striking discovery of the paper is that "forgotten" facts aren't actually erased. Using log-probability probes (Eq. 5.1), the author showed that even when a model fails every question about a fact, the weights still assign significantly higher probability to the original fact than they did before training.
"Forgotten facts keep most of the log-probability added by their write... later writes redirect the questions that reached it."
Think of it as a library where the books (the facts) are still on the shelves, but the librarian (the model's attention mechanism) has started giving everyone the same new bestseller regardless of what they asked for. In 70% of bare-statement failures, the model simply regurgitated the most recently learned fact.
Figure 2: Log-probability data showing that the "trace" of a fact remains in the weights long after the model stops being able to answer questions about it.
Mitigating Damage: The Frozen Teacher
To keep the model from losing its general intelligence (math, reasoning, etc.) during this constant weight-editing, the paper suggests Context Distillation from a frozen teacher.
- By penalizing the model when its overall output distribution drifts too far from the original "pure" model, researchers kept general abilities intact.
- However, even this didn't save the earlier facts. It saved the model's "brain," but not its "memory."
Critical Analysis & Future Outlook
This research draws a line in the sand for LLM architecture:
- Weights for Skills, Context for Facts: The weights are excellent for learning how to reason (skills), but they are an unreliable "system of record" for discrete facts.
- The Reachability Paradox: As we add more facts to weights, interference is inevitable. The "route" back to old info gets overwritten.
The Takeaway for Developers: If your application requires absolute reliability for composing and preserving facts over time, In-Context Learning (ICL) and RAG are superior to fine-tuning. The weights store the "content," but the context is what provides the "address" needed to find it.
Limitations
The study was conducted primarily on the Qwen3-4B/8B family using LoRA. While full fine-tuning showed similar results, the scale of "hundreds of writes" is still small compared to a lifetime of human learning. Future work needs to explore if larger models (70B+) possess more "address space" to mitigate this interference.
Conclusion
The model's weights are a crowded room. You can keep adding people (facts), and they might stay in the room, but eventually, everyone starts shouting over each other. To find the right person, you need a name tag—and in the world of LLMs, that name tag is the Prompt Context.