VGPO: Curing "Visual Amnesia" in Multimodal Reasoning Models
Visually-Guided Policy Optimization for Multimodal Reasoning
The paper introduces Visually-Guided Policy Optimization (VGPO), a reinforcement learning framework designed to enhance the reasoning capabilities of Vision-Language Models (VLMs). By utilizing an intrinsic "Visual Focus Score" and a progressive attention compensation mechanism, VGPO achieves state-of-the-art performance on multimodal reasoning benchmarks, including MathVista and MMMU-Pro.
TL;DR
Vision-Language Models (VLMs) often start a conversation by "looking" at the image but quickly "forget" it as they get lost in their own generated text. This paper introduces Visually-Guided Policy Optimization (VGPO), a reinforcement learning framework that uses a model's internal hidden states to refocus its attention on visual cues, effectively mitigating temporal forgetting and boosting reasoning accuracy by up to 33%.
The "Visual Forgetting" Crisis
Modern VLMs are remarkably good at logic, but they have a fatal flaw: they are text-centric by nature. Empirical analysis in this paper reveals a phenomenon called Temporal Visual Forgetting. As a model generates a long chain of thought (CoT), the attention weight assigned to visual tokens decays exponentially. By the time the model reaches the final "answer" phase, it is often hallucinating based on textual priors rather than looking at the actual image.
Existing solutions are often "heavy," requiring:
- Auxiliary Models: Using a second model (like GPT-5) to verify visual faithfulness.
- Additional Tokens: Forcing the model to output
<look_back>tokens. - Multiple Passes: Running the image through noised and clean versions to compare outputs.
Methodology: The "Visual Focus Score"
The authors ask a brilliant question: Can the model fix itself using its own internal signals?
They discovered that the cosine similarity between the hidden state of a generated token and a "visual prototype" (mean-pooled image embeddings) serves as a precise Visual Focus Score. When this score is high, the model is successfully "grounding" its thoughts in the image.
1. Visual Attention Compensation (VAC)
Since visual attention naturally decays, VGPO doesn't just reward visual focus—it progressively increases the requirement for it. As the reasoning chain gets longer, the model is incentivized more heavily to maintain high visual focus scores.
2. Dual-Grained Advantage Re-weighting
VGPO modifies the standard GRPO (Group Relative Policy Optimization) objective using two layers of rewards:
- Intra-trajectory (Token Level): Highlights specific tokens (like "the red circle") that show high visual activation.
- Inter-trajectory (Sequence Level): Compares different reasoning paths and favors the ones that maintained consistent visual grounding from start to finish.
Experimental Results: Slaying the Giants
VGPO was tested on the Qwen2.5-VL series. The results are striking. A 7B model tuned with VGPO can compete with or even beat 72B models on visual-heavy math tasks.
| Metric | Qwen2.5-VL-7B (Base) | + VGPO (Ours) | Relative Gain |
|---|---|---|---|
| Avg-Math | 50.0% | 66.6% | +33.2% |
| Avg-Vision | 48.7% | 63.3% | +30.0% |
Figure: Analysis showing that correct samples (right) maintain a much higher "Late/Early visual accumulation ratio" than incorrect ones.
Why It Works: Dynamic Grounding
The success of VGPO lies in its Linear Compensation Schedule. Unlike an exponential schedule which might over-correct or a step-function which is unstable for training, the linear approach matches the natural decay of attention. By "paying" the model more for visual focus in step 100 than in step 1, the optimizer forces the model to keep its "eyes" on the image throughout the entire logic chain.
Critical Analysis & Limitations
While VGPO is a breakthrough in efficiency (adding only ~5-10% overhead compared to the 20-30% of attention-weight-based methods), it has boundaries:
- The Perception Ceiling: If the visual encoder misinterprets an image (e.g., seeing a '6' as an '8'), VGPO will actually reinforce that error by forcing the model to focus on a flawed representation.
- Pure Logic Tasks: In scenarios where the final steps are purely arithmetic (e.g., "15 + 27"), mandated high visual focus might actually be a distraction.
Conclusion
VGPO proves that we don't always need "more data" or "bigger models" to solve hallucination. Sometimes, we just need to change the Policy Optimization to remind the model that the answer is right in front of its eyes. This "visually-guided" approach is likely to become a standard component in the fine-tuning of future multimodal agents.