BAMI: Deciphering and Fixing Inductive Bias in GUI Grounding Without Retraining

TL;DR

GUI Grounding—the ability for an AI to find and click the right button on a screen—is the "last mile" of GUI agents. However, even the best models often fail on professional, high-resolution software. BAMI (Bias-Aware Manipulation Inference) proves that these failures aren't due to a lack of "knowledge," but rather inherent Inductive Biases in how models process coordinates. By introducing a training-free recursive inference strategy involving hierarchical cropping and candidate re-selection, BAMI boosts SOTA performance by nearly 6% on the brutal ScreenSpot-Pro benchmark.

The "How" and "Why" of Grounding Failures

Why do multi-billion parameter models fail to click a simple icon? The authors conducted a pilot study using a new attribution method called Masked Prediction Distribution (MPD). By randomly masking parts of the screen and observing where the model "guesses," they categorized errors into two main types of Inductive Bias:

1. Precision Bias: MLLMs represent coordinates as discrete tokens (e.g., "789" as <7><8><9>). This discretization limits precision, often leading to offsets of several pixels—fatal on high-res screens.
2. Ambiguity Bias: Models are trained to minimize Edit Distance (text-based), but grounding requires minimizing Euclidean Distance (spatial). A model might think "189" is close to "789" because they share two digits, even though they represent opposite sides of the screen.

Methodology: Structured Inference

Instead of a simple one-shot prediction, BAMI uses a "Bias-Aware" manipulation loop:

1. Coarse-to-Fine Focus (Solving Precision Bias)

Mimicking human vision, BAMI first predicts a rough area, crops the screenshot to that region, and then re-predicts. This effectively increases the "relative resolution" for the transformer's attention mechanism, allowing it to see small UI elements more clearly.

2. Candidate Selection (Solving Ambiguity Bias)

To stop the model from falling into the "Edit Distance" trap, BAMI generates multiple candidates. Crucially, it uses a Masking Strategy: after the first prediction, it masks those pixels and asks the model to "find another." This forces the model to look at alternative semantic targets. An external correction model (like GPT-5 or a local Qwen-8B) then picks the best one using GUI-specific rules (e.g., "prioritize interactive icons over static text").

Experimental Results: The Training-Free Edge

BAMI was tested on ScreenSpot-Pro, a dataset of professional software (CAD, Scientific tools, etc.) where models usually perish.

• Consistent Gains: Accuracy jumped from 51.9% to 57.8% on TianXi-Action-7B.
• Model Agnostic: Whether using OS-Atlas or UI-TARS, BAMI provided a significant "test-time" performance boost.
• Ablation Success: Experiments showed that both Coarse-to-Fine and Candidate Selection are necessary; removing either one significantly drops the accuracy.

Critical Insight: The Power of Local Correction

One of the most impressive results is that BAMI doesn't require a massive "God-model" like GPT-5 to work. A locally trained Qwen3-VL-8B served as an effective correction model, achieving 56.2% accuracy. This suggests that the "knowledge" for correct grounding is already present in these smaller models; we just need a structured inference path to "remind" them of GUI priors.

Conclusion & Future Outlook

BAMI shifts the focus from "more data/more training" to "better inference." It highlights that current MLLM architectures have fundamental spatial blind spots caused by their token-based nature. While BAMI successfully patches these via test-time scaling, the next frontier will be designing model architectures where Euclidean spatial awareness is a core optimization objective, not just a side effect of text generation.

Takeaway: In the era of GUI agents, how you ask the model to reason (inference strategy) is just as important as how you trained it.