[ArXiv 2025] Reasoning for Recall: How RLVR Unlocks a Model's Hidden Knowledge

TL;DR

Can "thinking" help a model remember facts better? While we usually associate Chain-of-Thought (CoT) with math and logic, this paper proves that internal reasoning is a powerful tool for parametric knowledge retrieval. By applying Reinforcement Learning (RLVR) to factual QA, the authors boosted GPT-OSS-20B's performance on TriviaQA by +9.9%, showing that models have suppressed knowledge they don't know how to access by default.

Background: The Hidden Gap in Reasoning Models

We have entered the era of "reasoning models" (like OpenAI o1 or DeepSeek-R1), which are trained via RLVR to solve complex math and code. However, the academic community has largely ignored how these "thought tokens" affect fact retrieval.

The authors discovered a strange paradox: if you tell a model to "think step-by-step" on a math problem, it usually doesn't help because the model is already optimized for it. But if you give the same prompt for a trivia question, performance improves. This suggests that models are under-optimized for using their internal "thinking" space to navigate their own memory.

The Problem: Why "Just Thinking" Isn't Enough

Existing models suffer from two main issues:

1. Default Passive Retrieval: Models often try to "direct-fire" an answer for facts, missing the opportunity to leverage related concepts (Spreading Activation).
2. Unverifiable Traces: Unlike math, where $1+1=2$ is a hard logic path, knowledge retrieval is messy. How do we reward a model for "thinking correctly" about a fact?

Methodology: Training for Verifiable Factual Accuracy

The researchers proposed using RLVR (Reinforcement Learning from Verifiable Rewards) not for logic, but for truth.

The Reward Mechanism

They used a scalar reward function based on Natural Language correctness:

• 1.0 for an Exact Match (EM).
• 0.5 for Recall (the correct answer is contained within the prediction).
• A small penalty for formatting errors.

The Optimization Loop

Using a GRPO-style importance-sampling policy gradient, they trained GPT-OSS-20B on TriviaQA. Critically, the reward only cares about the final answer, allowing the model to develop whatever internal reasoning (interpretable or not) best elicits the memory.

Figure 1: Comparison of base performance vs. prompting vs. RLVR training, alongside a sample reasoning trace.

Experiments: Breaking the SOTA in Closed-Book QA

The results were striking. The RL-trained model didn't just get better at its training data (TriviaQA); it transferred its "memory access" skill to entirely different datasets.

Key Insight: SFT is not enough

The authors ran a baseline using Supervised Fine-Tuning (SFT) on correct traces generated by the base model. The RL-trained model significantly outperformed the SFT version, proving that the model needs the exploration of RL to find the optimal "mental paths" to its parameters.

Figure 2: Performance gains across multiple benchmarks including TriviaQA, NQ, and HotpotQA.

Critical Analysis: What is the Model Actually Doing?

Qualitatively, the reasoning traces became longer after RL. However, they aren't always "logical" the way a human would explain a fact. Sometimes the model "guesses" and iterates internally until the right memory provides a high-confidence signal.

The authors suggest this is a form of Spreading Activation: the model starts with a concept (e.g., "Canada's PM"), which activates related nodes ("Kim Campbell", "1993"), eventually leading to the correct recall.

Limitations

• Interpretability: The traces are often messy and not designed for human consumption.
• Hallucination Risk: While retrieval improved, the model may still "think its way" into a confident but wrong answer if the knowledge isn't in the parameters at all.

Conclusion: A New Era for RL in Factuality

This work shifts the narrative of reasoning models. "Thinking" isn't just for solving $x+y=z$; it's a retrieval mechanism. By training models specifically to use their reasoning tokens for memory access, we can create LLMs that are significantly more factual and reliable without increasing parameter counts.

Future Outlook: The next step is likely "Search-augmented" RLVR, where models learn to reason about when to trust their parameters versus when to use an external search engine.