TRICE: Elevating Thinking Models with a Full-Pipeline Tool-Integration Recipe
Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning
This paper introduces TRICE, a full-pipeline recipe for teaching "thinking" models (long-CoT LLMs) to perform Tool-Integrated Reasoning (TIR) by interleaving natural language and code execution. Developed for Qwen3 4B/30B scales, TRICE achieves state-of-the-art results, including 99.2% on AIME 2025, by balancing supervised fine-tuning (SFT) and reinforcement learning (RL) with verifiable rewards.
TL;DR
While "thinking" models (like o1 or DeepSeek-R1) have mastered textual reasoning, they often struggle with formal computation and combinatorial search. TRICE (Textual Reasoning Interleaved with Code Execution) provides the first comprehensive "recipe" to bake tool-use into these models without breaking their internal logic. By optimizing the transition from SFT to RL and leveraging a stateful sandbox, TRICE-30B achieves an incredible 99.2% on AIME 2025, proving that a 30B model with a tool can outperform a 200B+ model without one.
The "Thinking Model" Paradox: Why Tools Often Make Them Worse
It seems intuitive: give a smart model a calculator, and it should get smarter. However, the authors observed a paradox: when thinking models are given tool access, their performance often degrades, even if they don't actually call the tool.
This happens because the "tool prompt" acts as a distractor that disrupts the fragile chain-of-thought (CoT) logic. Models either fall into the "delayed-code pattern" (writing code only at the very end to verify an answer they've already guessed wrong) or start "hallucinating" code execution results in plain text when no tool is available, leading to a catastrophic collapse of native reasoning.
Methodology: The TRICE Recipe
The researchers didn't just add more data; they re-engineered the entire training pipeline.
1. Data Engineering: The "Learnability" Over Accuracy
The authors found that a teacher’s accuracy isn't the best metric for SFT. Instead, learnability matters. They compared heavyweight teachers (long, self-contained scripts) with lightweight teachers (short, incremental snippets). The lightweight style allowed the student to "think" turn-by-turn.
- Tool-Advantaged Problems: They filtered for problems where tool-use actually provided a performance delta over text-only reasoning, ensuring the student learns when a tool is actually useful.
- The Mixed-Data Strategy: To prevent models from forgetting how to think without code, they mixed TIR trajectories with pure text-only trajectories.
2. The Three Stages of TIR SFT
A critical insight of this paper is the Form → Substance → Noise progression:
- Stage 1 (Form): The model learns the syntax of calling a tool but does it poorly, leading to loops and errors.
- Stage 2 (Substance): The model begins to understand how to use the tool output to guide the next reasoning step. This is the "Goldilocks" zone for RL initialization.
- Stage 3 (Noise): The model starts overfitting to teacher-specific quirks (like specific response lengths), causing RL stability to plummet.
Figure: Monitoring Pass@k and Response Length is crucial for identifying the RL-ready checkpoint.
3. Stable RL with Verifiable Rewards (RLVR)
To fine-tune the final policy, they used RLVR. Crucially, they found that standard off-policy updates caused training collapse due to the unpredictable nature of tool returns. By switching to fully on-policy training and using Rollout Routing Replay, they achieved a stable boost in performance.
Experiments: SOTA Achievement
TRICE was tested on the most rigorous math benchmarks available, including the fresh AIME 2025 and APEX 2025.
| Model | AIME 2025 | HMMT 2025 | Avg. Performance | | :--- | :---: | :---: | :---: | | Qwen3-30B Base | 88.8% | 75.6% | 67.1% | | TRICE-30B (TIR) | 99.2% | 92.5% | 81.9% |
Table: TRICE-30B dominates the ~30B scale and rivals frontier closed models.
Deep Insight: Code as a "Cognitive Tool"
The most profound takeaway is that the model stopped using Python as just a calculator. As shown in the classification analysis, the code executor served four roles:
- Empirical Discovery: Testing small numbers to find a pattern.
- Algorithmic Search: Running a brute-force search when the math is too complex for CoT.
- Computation Offloading: Handling the "grunt work" of arithmetic.
- Conjecture Verification: Checking if an intermediate theorem holds true.
Conclusion
The TRICE recipe demonstrates that the future of AI reasoning isn't just about "more parameters," but about "better tools and better integration." By treating the interaction between language and code as a first-class citizen in the training pipeline, we can create models that are not only more accurate but also more efficient, using code to compress thousands of manual reasoning tokens into a few lines of execution.
Critical Analysis: While highly effective in math and science, the reliance on verifiable rewards (pass/fail) means this recipe is currently restricted to domains with "ground truth." Extending TRICE to subjective "agentic" tasks (like web browsing or coding architecture) remains the next great frontier.