[Research Insight] START: Breaking "Thinking Stagnation" in General QA via Decoupled RL
RLVR Training of LLMs Does Not Improve Thinking Ability for General QA: Evaluation Method and a Simple Solution
This paper investigates Reinforcement Learning from Verifiable Rewards (RLVR) in the context of General Question Answering (GQA). It introduces a Cross-Generation evaluation framework and a two-stage training method called START (Separated Thinking And Response Training), achieving a 59.24% win rate over vanilla GRPO on GQA benchmarks.
TL;DR
Is the "thinking" process in models like DeepSeek-R1 actually useful for general tasks, or is it just a byproduct of math training? This paper reveals a discouraging reality: standard RL on General QA (GQA) leads to Thinking Stagnation, where models learn to provide "better-sounding" answers without improving their internal logic. To fix this, the authors introduce START, a two-stage training paradigm that forces the model to evolve its "thinking engine" by masking gradients during the answering phase.
The "Thinking" Disparity: Reasoning vs. General QA
The current AI trend is to let models "think" before they speak. In verifiable domains like MATH or Code, this works perfectly because a correct answer requires a solid logical chain. However, in General QA (e.g., "How do I plan a trip?"), the link is weaker.
The authors use a Cross-Generation framework to prove this:
- In MATH: A high-quality thinking trace from an 8B model can make a tiny 1.7B model perform nearly as well as the 8B model itself. The thinking is the bottleneck.
- In GQA: A better thinking trace provides only marginal gains. The "answering head" (the style and phrasing) actually contributes more to the win rate than the logic behind it.
Problem: The Reward Shortcut
In general domains, Reward Models (RMs) often prefer longer, more polite, or better-structured answers. During standard RL (like GRPO), the model realizes it can get higher rewards by simply tweaking the final answer tokens. This is a shortcut. The model stops "evolving" its thinking because thinking is hard, but changing the style of the response is easy. The authors call this Thinking Stagnation.
Methodology: START (Separated Thinking And Response Training)
To break this stagnation, the authors propose a conceptual shift: Treat the answering phase as part of the environment, not the agent.
Phase I: Thinking Evolution
The model generates both a thinking trace () and an answer (). However, during backpropagation, the gradients for tokens in are masked (zeroed out).
- The model cannot change how it answers to get more reward.
- The only way to increase reward is to provide a "thought" that is so useful it makes the existing answering head produce a better result.
Phase II: Joint Optimization
Once the "thinking engine" is upgraded, the mask is removed. The model now learns to align its response with its new, deeper cognitive abilities.
Experimental Triumphs
Using Qwen3-1.7B and ExpertQA, the authors compared START against standard GRPO and GRPO-MA (a multi-answer baseline).
| Metric | Vanilla GRPO | GRPO + START |
|---|---|---|
| Reward (Final) | 0.2182 | 0.2201 |
| Win Rate vs Base | - | 59.24% |
| Thinking Utility (Tpost + Apre) | Base | 68.15% (Win Rate) |
The "Tpost + Apre" result is the most critical: it shows that the thinking traces generated by START are inherently more valuable to a completely different, frozen model than the ones generated by standard training.
Deep Insight: The Emergence of "Meta-Context"
Why does START work? Qualitative analysis shows that START forces the model to use its "thinking space" for Meta-Context Modeling.
- User Persona: The model starts explicitly guessing who the user is (e.g., "The user seems to be a student") to calibrate its technical level.
- Structural Planning: The model outlines how it will organize the response before writing it.
In standard RL, these behaviors are rare because the model just "blurt out" a good-sounding answer. START makes "thinking" functional again.
Discussion & Limitations
While START is an "out-of-the-box" solution for any RL framework (DAPO, GRPO), it does rely on a two-phase schedule, which adds some complexity to the training pipeline. Furthermore, while it fixes the "loose coupling" in GQA, the field still lacks a definitive "verifiable" reward for general creativity or helpfulness, leaving us dependent on Reward Models that can still be gamed—though START makes that gaming much harder.
Conclusion
This paper provides a wake-up call for the "Thinking Model" era: Thinking doesn't happen for free. For General QA, we must architect our training to prevent models from taking the path of least resistance. START offers a simple, elegant mechanism to ensure that when an LLM "thinks," it’s actually doing work, not just wasting tokens.