[ArXiv 2025] Stable Adaptive Thinking: Solving the LRM Overthinking Crisis via Advantage Shaping

TL;DR

Large Reasoning Models (LRMs) like O1 and DeepSeek-R1 are brilliant but expensive—they often write a "novel" to solve "2+2". This paper presents a framework to make LRMs difficulty-aware. By introducing Correctness-Preserving Advantage Shaping (CPAS) and Length-Aware Gradient Regulation (LAGR), the authors created a model that knows when to think deep and when to answer instantly, boosting accuracy by ~3.7 points while cutting 44% of token costs.

The "Overthinking" Tax and Optimization Instability

The current paradigm of LRMs relies on extended reasoning traces (Chain-of-Thought). However, two "hidden killers" plague adaptive reasoning research:

1. Exploration Suppression: Standard RL rewards short answers. This creates a bias where a correct but long reasoning path gets a negative advantage compared to a short one, teaching the model that "thinking is bad."
2. Gradient Dilution: In a single training batch, one response might be 100 tokens while another is 10,000. In standard GRPO, the long response's gradient is so diluted that the "Mode Switch" token—the most important part of the decision—is ignored.

Methodology: The Two-Stage Scalpel

Stage 1: Hybrid Fine-Tuning (HFT)

The model is first "warmed up" on a balanced diet of /think (long CoT) and /no_think (direct answer) data. This ensures the model can perform both behaviors before it is asked to choose between them.

Stage 2: Stable RL Training

This is where the core innovation happens via two mechanisms:

• CPAS (The Safety Net): Instead of just rewarding brevity, CPAS ensures that as long as a long-chain response is correct, it isn't penalized into oblivion. This preserves the model's "deliberative capacity" for hard problems.
• LAGR (The Balancer): It applies a weight $w_i^{(\beta )}$ to gradients. By setting $\beta \approx 0.4$, the authors balance the optimization contribution of short vs. long responses. Crucially, they apply a boosting factor ($\lambda$) to the prefix control token to ensure the "Decision to Think" is updated with high priority.

Figure 1: The two-stage pipeline: HFT for capability, then RL with CPAS and LAGR for adaptive decision making.

Experimental Battleground

The model was tested on brutal math benchmarks: MATH-500, AIME-2024, and AIME-2025.

• Efficiency: On MATH-500, the model used the no_think mode for 77.6% of queries.
• Accuracy: Unlike previous "efficiency-focused" models that lose accuracy, this method actually increased it. By not forcing a long CoT on trivial problems (which can introduce hallucinations), the model stayed more robust.

Table 1: Comparison of Qwen2.5-7B across benchmarks. Note the massive reduction in average length ($\Delta$Len) while improving Acc.

Deep Insight: Difficulty-Awareness

A fascinating result from the analysis (Figure 3 in the paper) shows a direct correlation between problem difficulty and the "No-Thinking Ratio." For Level 1 MATH problems, the model rarely thinks. For Level 5 and AIME problems, it automatically shifts gears into high-compute "Thinking Mode." This is the "System 1 vs. System 2" duality finally realized in a stable training loop.

Figure 2: The model naturally scales thinking length as query difficulty increases.

Critical Perspective & Conclusion

This paper solves a major engineering headache in training LRMs: the instability of mixed-length RL. By treating the "Control Token" as a privileged signal and protecting correct long-form reasoning, it provides a blueprint for making models like DeepSeek or O1 significantly cheaper to run without losing their "intelligence."

Limitations: The study is currently centered on 1.5B and 7B models. Whether these gradient regulation scales hold for 70B+ parameters remains the next frontier. Furthermore, the reliance on verifiable (math) rewards means applying this to creative writing or open-ended reasoning will require high-quality Reward Models (RMs).

Final Takeaway: Thinking longer is compute-intensive; thinking smarter about when to think is the true path to efficient AGI.