[CVPR 2025/ArXiv] GUI-Libra: Taming the "Reasoning-Grounding" Conflict in Native GUI Agents

TL;DR

Building a native GUI agent involves more than just teaching a VLM to click. GUI-Libra addresses two critical "post-training" bottlenecks: the tendency for long reasoning (CoT) to distract the model from precise clicking (grounding), and the instability of RL when multiple "correct" paths exist. By introducing Action-aware SFT and Conservative RL, the team from Microsoft and UIUC has pushed 4B-8B models to outperform much larger 70B+ systems.

The "Thinking Too Much" Trap: Motivation

In typical VLMs, we want long Chain-of-Thought (CoT) to improve reasoning. However, as the authors discovered, for GUI agents, longer reasoning tokens often correlate with worse grounding.

Why? Because the grounding tokens (coordinates) represent only a tiny fraction of the sequence; they get "lost" in the loss function dominated by verbose thoughts. Furthermore, in RL for GUIs, we face Partial Verifiability: if a human demonstration clicks a button, a model clicking a different (but equally valid) link is penalized as "wrong." This ambiguous negative feedback leads to unstable training and "reward hacking."

Methodology: The GUI-Libra Recipe

1. Action-Aware SFT (ASFT)

To solve the interference between reasoning and grounding, the authors don't just dump data into the model. They use a mixed-supervision approach:

• Mixed Data: Training on both Reasoning -> Action and Direct Action pairs.
• Token Reweighting: They mathematically boost the loss weight for action and grounding tokens (${\alpha }_a$ and ${\alpha }_g$), ensuring the model prioritizes where to click over how to explain it.

2. Conservative RL (KL + SNGS)

Traditional RLVR (RL from Verifiable Rewards) often drops KL regularization for speed. GUI-Libra argues this is a mistake for multi-step tasks.

• KL Trust Region: Keeps the model from drifting into "unfamiliar" UI states where offline metrics can no longer predict online success.
• Success-adaptive Negative Gradient Scaling (SNGS): If the model finds an action that doesn't match the demo but the group success rate is high, the negative gradient is downweighted. This prevents the model from being over-penalized for valid alternative paths.

Experimental Breakthroughs

The results on AndroidWorld and WebArena are striking. GUI-Libra-8B (based on Qwen) matches or beats GPT-4o and GPT-5-mini when they are used within standard agent frameworks.

One of the most profound findings is the Offline-to-Online Correlation. By using KL regularization, the authors showed that offline "step accuracy" becomes a much more reliable predictor of whether the agent will actually finish a task in a live environment.

Critical Insight & Conclusion

GUI-Libra proves that "Native" agents (one single model for everything) are viable and can surpass "Modular" agents (VLM + external planner). The secret isn't just more data—it's data alignment and conservative optimization. By acknowledging that GUI agents operate in a world where "there's more than one way to do it," GUI-Libra sets a new standard for robust, reasoning-capable agents.

Future Outlook: The next step is scaling this to fully online, interactive training where the agent explores the web in real-time without solely relying on static, partially-verifiable demonstrations.