RAG over Thinking Traces: Turning Process into a Reusable Resource
RAG over Thinking Traces Can Improve Reasoning Tasks
This paper introduces T3 (Transformation of Thinking Traces), a novel RAG framework that retrieves intermediate "thinking trajectories" instead of standard documents to solve reasoning-intensive tasks. By retrieving and transforming these traces into structured forms, the method achieves significant SOTA gains, such as a +56.3% relative improvement for Gemini-2.5-Flash on AIME.
TL;DR
While Retrieval-Augmented Generation (RAG) is the gold standard for factual Q&A, it has long been dismissed as "useless" for heavy reasoning tasks like math. This paper from UC Berkeley flips the script: the problem isn't RAG—it's the corpus. By retrieving Thinking Traces (the intermediate steps of a model's logic) and structured "how-to" guides instead of static Wikipedia pages, models like GPT-5 and Gemini-2.5-Flash see massive performance leaps (up to +56%) with even lower inference costs.
The "Reasoning Gap" in Traditional RAG
Traditional RAG relies on retrieving "facts." But solving an AIME math problem or a complex coding challenge isn't about knowing a fact; it's about following a process.
Current models struggle with reasoning RAG for two reasons:
- Corpus Alignment: Web documents contain solutions, but rarely the step-by-step cognitive "detours" or "reflections" needed to navigate a new, similar problem.
- Noise & Verbosity: Raw thinking traces from "O1-style" models are often thousands of tokens long, filled with "wait, let me re-check that" loops that confuse the solver model.
Methodology: T3 (Transformation of Thinking Traces)
The authors propose a simple yet profound shift: Treat the "thinking" of strong models as a reusable library.
The T3 Pipeline
- Trace Collection (Offline): Strong models (e.g., Gemini-2-thinking, QwQ-32B) solve an auxiliary set of problems, generating long-form thinking traces.
- Transformation (Offline): A smaller, cheaper model "rewrites" these traces into three optimized formats:
- Struct: Turns raw rambling into a clean, procedural scaffold.
- Semantic: Distills the core "aha!" moment and central insight.
- Reflect: Focuses on "what not to do" by identifying traps and pitfalls.
- Retrieve-then-Generate (Inference): When a new problem arrives, the system retrieves the top-k most relevant transformed traces to guide the solver.
Experimental Proof: Better Logic, Lower Price
The results on AIME (Competition Math), GPQA (Science), and LiveCodeBench (Coding) are startling.
- Performance Surge: For Gemini-2.5-Flash, AIME accuracy jumped from 53.3% to 83.3% when using semantic distillation.
- Cost Efficiency: In the case of GPT-5, using T3 actually reduced the cost per query by 15%. Why? Because the retrieved "hint" allowed the model to arrive at the answer faster, spending fewer tokens on internal "thinking."
Key Insight: Trace Quality > Corpus Size
One of the paper's most critical findings is that who thinks matters. 59k traces from Gemini-2-thinking consistently outperformed 114k traces from QwQ-32B. High-quality logic is a more potent retrieval unit than a massive volume of mediocre steps.
Critical Analysis & Conclusion
This work signals a shift in how we build AI systems. Instead of purely "scaling" internal parameters or "scaling" test-time compute (search), we can scale externalized experience.
Limitations:
- The current corpora are heavily math-skewed.
- It uses "vanilla" RAG; more advanced iterative RAG might unlock even more.
The Takeaway: Thinking is no longer a ephemeral process that disappears after EOF. By storing, transforming, and retrieving "thinking traces," we can help models stand on the shoulders of their (and others') previous cognitive efforts.