ICR: Solving the Overthinking Dilemma in LLM Reasoning via Implicit Regularization
Implicit Compression Regularization: Concise Reasoning via Internal Shorter Distributions in RL Post-Training
This paper introduces Implicit Compression Regularization (ICR), an on-policy regularization method designed to mitigate "overthinking" in LLM reasoning. By reinforcing the shortest correct responses within rollout groups during Reinforcement Learning with Verifiable Rewards (RLVR), ICR achieves significant token reduction while preserving or even enhancing SOTA reasoning accuracy across mathematical and knowledge-intensive benchmarks.
Executive Summary
TL;DR: Reasoning models often "overthink," generating bloated chain-of-thought traces that waste compute and sometimes introduce errors. Implicit Compression Regularization (ICR) is a novel RL post-training strategy that identifies the shortest correct reasoning paths a model is already capable of producing and encourages the policy to favor them. Unlike traditional length penalties that "punish" tokens, ICR "rewards" efficiency, resulting in faster inference without the typical accuracy drop.
Academic Positioning: This work moves beyond heuristic reward-shaping (like Kimi-k1.5's length penalties) by providing a formal definition of overthinking based on group-wise correlation and introducing a data-driven regularization approach that stays on-policy.
The Problem: The Cost of "Thinking Too Much"
Reinforcement Learning with Verifiable Rewards (RLVR) has been remarkably successful in teaching models to "think." However, we are currently in an era of Overthinking. Models frequently repeat self-reflections, engage in redundant verification, or get lost in "circular reasoning" just to maximize the probability of a correct answer.
Existing fixes are flawed:
- Length Penalties (LP): They treat response length as a target. This often leads to Underthinking, where the model cuts off logic prematurely to avoid the penalty, tanking accuracy.
- Early Exit/Truncation: These methods assume the end of a reasoning trace is useless. As shown in the paper's experiments (Fig 1), hard truncation kills accuracy in dense math problems.
Insights: The Correlation Switch
The authors discovered a fascinating dynamic: in the early stages of RL, shorter responses are actually more likely to be correct within a rollout group (negative correlation). As training continues with standard penalties, this correlation becomes positive—meaning the model has been "pushed" so hard that now only long responses have a chance at being right. That is the transition from overthinking to underthinking.
Methodology: Reinforcing the "Aha!" Moments
ICR doesn't add a negative reward for length. Instead, it modifies the objective to give an extra "boost" to the shortest correct samples within a group.
The Formal Objective
Instead of just maximizing (correctness), ICR adds a term: Where is the set of the shortest correct samples in the rollout.
Note: The mechanism relies on identifying the induced "virtual shorter distribution" within the existing on-policy rollouts.
Why It Works
By only looking at correct samples, ICR ensures that the "compression signal" comes from valid reasoning. It effectively tells the model: "You found three ways to solve this; the two-paragraph version was correct, so let's do more of that and less of the five-paragraph version."
Experimental Results: Breaking the Pareto Frontier
The results across Qwen3 and DeepSeek-Distill models are striking. In mathematical reasoning tasks (AIME, GSM8K), ICR achieved:
- Significant Compression: Shorter traces compared to standard GRPO.
- Accuracy Gains: Unlike length penalties which degrade performance, ICR often improved accuracy because it eliminated "noisy" or "distractor" reasoning steps.
Figure: The Pareto frontier shows that ICR (red line) stays higher (more accurate) for every given response length compared to traditional length penalties.
Deep Insight & Conclusion
The core takeaway is that redundancy is a symptom of poor policy alignment, not a requirement for intelligence. ICR proves that the "shorter path" to a solution usually exists within the model's latent capabilities; we just need a surgical way to extract it without breaking the underlying logic.
Limitations: While ICR is excellent at preserving accuracy, it is a "soft" signal. If a developer needs a strict token budget (e.g., "never more than 512 tokens"), ICR still needs to be paired with a mild length penalty to enforce a hard ceiling.
Future Outlook: This approach paves the way for "Lossless Reasoning Compression," where LLMs can scale their "thinking" dynamically based on task difficulty rather than defaulting to maximum verbosity.