[CVPR 2025] VGPO: Mitigating "Temporal Visual Forgetting" in Multimodal Reasoning
Visually-Guided Policy Optimization for Multimodal Reasoning
The paper introduces Visually-Guided Policy Optimization (VGPO), a reinforcement learning framework designed to enhance the multimodal reasoning of Vision-Language Models (VLMs). By leveraging intrinsic hidden states for visual grounding and implementing a dual-grained advantage re-weighting strategy, VGPO achieves state-of-the-art results, including +33.2% gain on mathematical reasoning and +30.0% on vision-dependent tasks.
TL;DR
Even the most advanced Vision-Language Models (VLMs) often "forget" the image halfway through a long math problem. Visually-Guided Policy Optimization (VGPO) fixes this by using the model's internal hidden states to detect visual focus and explicitly rewarding the model for staying "visually faithful" throughout the entire reasoning process.
Background: The Invisible Bottleneck
While Reinforcement Learning with Verifiable Rewards (RLVR) has significantly boosted the logical capabilities of models, a silent killer remains: Text-Dominated Inference. As models generate longer "Chain-of-Thought" steps, their attention to the input image tokens decays—a phenomenon the authors label Temporal Visual Forgetting.
Figure: The red line (visual attention) drops sharply as the reasoning steps (blue/gray lines) grow, leading to incorrect samples (left histogram).
The Core Insight: Hidden States as a Compass
Most current solutions rely on "Look-Back" tokens or auxiliary models to verify visual grounding. VGPO takes a more elegant, "zero-dependency" approach. It discovers that the cosine similarity between a generated token's hidden state and the image's "visual prototype" serves as a precise Visual Focus Score.
Why is this a big deal? It means the model internally knows when it is looking at the image; it just needs a training signal to keep doing it.
Methodology: Visually-Guided Policy Optimization
VGPO introduces three innovations to the standard RL (GRPO) pipeline:
- Visual Focus Score: Measuring how much each generated token aligns semantically with the image.
- Visual Attention Compensation (VAC): A linear schedule that increases the intensity of the reward for visual focus as the sequence gets longer. This acts as a counterweight to natural decay.
- Dual-Grained Re-weighting:
- Intra-trajectory: Rewards specific tokens that ground their logic in visual cues.
- Inter-trajectory: Favors entire reasoning paths that maintain consistent visual contact over those that just "guess" based on text.
Figure: The VGPO workflow: from hidden state similarity to dual-level advantage re-weighting.
Performance: 7B Models Punching Above Their Weight
The results are striking. By simply making the 7B model more "attentive" to the image, it achieves results that rival 72B parameter giants.
- MathVista Accuracy: Topped at 74.1%.
- MMMU-Pro (Vision-Only): Significant jumps (+30% relative gain) indicating the model isn't just getting better at math, but better at seeing.
- Stability: Unlike standard RL which can be "swingy," VGPO shows a much smoother reward curve by providing denser, vision-aligned feedback.
Table: VGPO outperforms SOTA methods like PAPO and VPPO across nearly all benchmarks.
Critical Perspective: Is "Looking" Enough?
The authors provide a refreshingly honest "Failure Case Analysis." They note that VGPO makes the model look closer, but if the underlying Visual Encoder is weak (e.g., misidentifying a blurry '6' as an '8'), VGPO will actually reinforce that mistake.
Takeaway: VGPO optimizes the utilization of visual evidence. It ensures the model doesn't drift away into textual "hallucination," but it cannot fix a model that is essentially "nearsighted" at the sensor level.
Conclusion
VGPO marks a shift from treating VLMs as black boxes to using their internal "semantic signals" for better RL. By solving temporal forgetting, it paves the way for reliable, long-form multimodal assistants that won't lose sight of the facts.