[CVPR 2025] HART: Breaking Resolution Constraints in LMMs via Annotation-Free RL

TL;DR

Current Large Multimodal Models (LMMs) are often "blinded" by high-resolution inputs, struggling with an explosion of redundant tokens. HART (High-resolution Annotation-free Reasoning Technique) introduces a closed-loop reinforcement learning framework that teaches models to "zoom in" on critical image regions. Remarkably, it requires zero human-annotated bounding boxes, yet allows a 7B model to outperform 72B giants on vision-centric benchmarks by self-verifying its own visual attention.

The Motivation: The "Right Answer, Wrong Reason" Trap

In high-resolution tasks (like reading a tiny sign in a 4K street view), most visual tokens are noise. While we want models to first "ground" (locate) the target and then "reason," training this usually requires expensive human labels.

The authors identified a critical flaw in existing RL-based grounding: Reward Misspecification. In pilot tests with Qwen2.5-VL and InternVL3, they found that even when models gave the correct answer, their internal "eye" was looking at the wrong part of the image 58% to 90% of the time. Essentially, the models were "guessing" correctly based on context rather than seeing.

Methodology: HART and the Power of Self-Verification

HART acts like a strict teacher. It follows a two-step process:

1. Grounding: The model predicts a Region of Interest (ROI) based on a low-res version of the image.
2. Verifiable Reasoning: The full image is hidden. The model is given only the high-res crop of its predicted ROI.

If the model can answer the question correctly using only that crop, it proves the grounding was "faithful." This creates a "closed-loop" where the answer correctness serves as a direct proxy for grounding quality.

AP-GRPO: Better Math for Better Sight

To stabilize this, the authors improved the Group Relative Policy Optimization (GRPO) algorithm. Their version, AP-GRPO (Advantage Preference GRPO), adds a dynamic weight ($k$). It prioritizes updates for samples where both grounding and reasoning are correct, effectively filtering out the "lucky guesses" that plague standard RL training.

Experiments: Small Model, Huge Results

HART was tested on grueling high-resolution benchmarks like MME-RealWorld and TreeBench.

• SOTA Performance: HART-7B achieved 62.4% on MME-RealWorld-Lite. For context, Qwen2.5-VL-72B (10x larger) only scored 43.7%.
• Domain Gains: The model saw massive jumps in specialized fields: +26.0% in Remote Sensing and +27.7% in Autonomous Driving perception.
• Grounding Accuracy: Even without box-labels during training, HART's grounding (IoU) outperformed the best previous RL methods by over 5.5%.

Visualization: Seeing is Believing

The difference is clear when looking at the model's "thought process." In tasks requiring identification of specific people or small objects, base models like InternVL3 often focus on the center of the image (a common bias), whereas HART successfully shifts its focus to the "right-most man" or specific text cues required by the prompt.

Critical Insights & Future Work

HART proves that Interpretable Reasoning Pathways are more valuable than raw parameter counts. By forcing the model to "explain" its answer through a specific visual crop, we get both better performance and a window into why the model made its decision.

Limitations:

• Inference Speed: The two-turn "ground-then-reason" process is slower than a single pass.
• Scale: The current work focuses on 7B-8B models. Scaling this to 70B+ models could potentially unlock even higher levels of visual intelligence.

In conclusion, HART is a significant step toward "Human-like" vision for AI: moving from processing every pixel to intelligently glancing at what matters.