[arXiv 2025] FIPO: Breaking the Reasoning Ceiling with Future-KL Dense Supervision

TL;DR

The Qwen Pilot Team has released FIPO (Future-KL Influenced Policy Optimization), an RL algorithm that finally solves the "uniform credit assignment" problem in GRPO-style training. By re-weighting tokens based on their downstream impact (Future-KL), they enabled Qwen2.5-32B to scale its internal "thinking" from 4k to 10k+ tokens, surpassing o1-mini and DeepSeek-R1-Zero on AIME 2024 benchmarks.

The Problem: The "Flat Reward" Trap

In standard RLVR (Reinforcement Learning with Verifiable Rewards), we often use GRPO. It’s efficient because it doesn't need a Critic/Value model. However, it has a fatal flaw: it treats every token in a correct trajectory as equally "good."

Whether a token is a brilliant logical leap or a simple "the," it receives the same advantage score. This results in:

1. Length Stagnation: Models stop exploring deeper reasoning paths.
2. Coarse Credit Assignment: The model can't identify which specific step led to the correct answer.

Methodology: The Intuition of Future-KL

FIPO's core insight is that the value of a token is defined by the trajectory it initiates.

1. Probability Shift ($\Delta \log p$)

The authors track the difference between the current policy and the old policy. A positive shift means the model is leaning into a specific token.

2. Future-KL Formulation

Instead of just looking at the current token, FIPO calculates the Future-KL: the cumulative, discounted sum of future probability shifts.

$FutureK{L}_t={\sum }_{k=t}^T{M}_k\cdot {\gamma }^{k-t}\cdot \Delta \log p_k$

This acts as a high-fidelity signal: if a token leads to a future sequence that the model is increasingly confident in, that token gets a massive boost.

Figure: Performance of FIPO vs Baselines on AIME2024.

3. Stability via Masking and Clipping

To prevent the exponential nature of KL from exploding, FIPO introduces:

• Extreme Value Filtering: Masking tokens that exceed safety thresholds.
• Soft Decay Window: A "half-life" parameter ($au$) that ensures the model focuses on the local logical chain rather than the distant, noisy future.

The Scaling Effect: Length is a Proxy for Depth

One of the most striking results is the correlation between CoT length and performance. FIPO doesn't just make the model "wordy"; it elicits Self-Reflection.

Figure: As training progresses, FIPO pushes the entire distribution of response lengths upward, which directly correlates with higher AIME accuracy.

The paper identifies four stages of reasoning evolution:

• Stage 1: Superficial Planning (Brief, hallucinated conclusions).
• Stage 2: Linear Execution (Standard CoT, single-pass logic).
• Stage 3: Emergent Self-Reflection (Proactive verification of intermediate steps).
• Stage 4: Systematic Deep Reasoning (Multi-pass auditing and granular arithmetic verification).

Experimental Results

FIPO was tested on the Qwen2.5-32B-Base model with remarkable results:

• AIME 2024 Pass@1: 56.0% (Peak 58.0%), compared to DAPO's 50.0%.
• CoT Length: Scaled from 4,000 to over 10,000 tokens.
• Comparison: Outperforms o1-mini (~56.0%) and DeepSeek-R1-Zero-Math-32B (~47.0%).

Critical Analysis & Conclusion

FIPO proves that dense supervision is possible without the overhead of PPO's Critic models. By mathematically formulating the "downstream influence" of a token, we can bridge the gap between pure outcome-based rewards and step-by-step reasoning.

Limitations:

• Inference Cost: 10,000+ tokens per query is computationally expensive.
• Domain Specificity: The current success is primarily in Mathematics, where rewards are strictly verifiable.

Future Outlook: FIPO provides a blueprint for "System 2" thinking in models. The next frontier will be distilling these 10k-token "deep thoughts" back into efficient, shorter responses without losing the underlying logical rigor.