VisBrowse-Bench is a new expert-constructed benchmark designed to evaluate Multimodal Large Language Models (MLLMs) in "visual-native" browsing tasks. Covering 169 VQA instances across seven domains, it enforces multi-hop reasoning where visual evidence is indispensable, revealing that even top-tier models like Claude-4.6-Opus and o3-deep-research achieve less than 48% accuracy.
TL;DR
Researchers from CASIA, Ant Group, and several top universities have released VisBrowse-Bench, a diagnostic benchmark that exposes a critical weakness in current Multimodal Large Language Models (MLLMs): they are largely "blind" during the search process. While models like Claude-4.6 and o3 can find text, they struggle to discover and reason with visual evidence that isn't explicitly handed to them. Even the best models fail more than half the time on this dataset.
The "Textual Shortcut" Crisis
Most current "multimodal" benchmarks are multimodal in name only. If an agent can turn an initial image into a text caption (e.g., Identifying "Iron Man Poster"), the rest of the task usually becomes a standard text-based search.
The authors of VisBrowse-Bench argue that real-world browsing is visual-native. Information is often embedded in images, diagrams, or spatial layouts that have no textual equivalent. Existing models act as "tool-callers" rather than "visual-thinkers," relying on internal knowledge or text snippets while ignoring the rich visual context of the web.
Methodology: Enforcing Visual-Native Reasoning
To break the reliance on text, VisBrowse-Bench was built on two non-negotiable pillars:
- Multimodal Integration: Reasoning chains must involve at least three "hops" and two distinct pieces of visual evidence.
- Visual Competency: Questions are designed so that spatial grounding (e.g., "the person at the top right") and pixel-level analysis are the only ways to proceed.
The Agentic Workflow
The researchers didn't just provide data; they provided a benchmark for active agents. Their workflow equips MLLMs with five core tools:
- Text/Image Search: For standard retrieval.
- Reverse Image Search (RIS): To identify unknown entities.
- Image Crop (IC): For fine-grained local reasoning.
- Webpage Visit: To parse structured info.
Figure: The VisBrowse-Bench design ensures that visual capabilities are essential, not optional.
Experimental Analysis: A Reality Check for SOTA
The results are a humbling wake-up call for the industry. Even with the full suite of tools (+IS), performance remains surprisingly low:
| Model | Direct Answer (Zero-shot) | + Image Search (+IS) | | :--- | :---: | :---: | | Claude-4.6-Opus | 27.2% | 47.6% | | o3-Deep-Research | 41.1% | N/A | | Kimi-K2.5 | 21.3% | 41.4% | | Qwen3-VL-Plus | 17.8% | 32.5% |
Key Insights:
- Parametric Knowledge is Not Enough: Under "Direct Answer," most models stay below 30%. They simply don't have the "long-tail" facts required for deep research.
- The Claude vs. o3 Paradox: While o3-Deep-Research has higher raw reasoning capability (41% direct), Claude-4.6-Opus's superior tool-use integration allows it to pull ahead when given a browser.
- Tool Usage Signatures: Top models like Claude-4.6-Opus show a balanced usage of all 5 tools, including frequent cropping (18.3%), whereas weaker models over-rely on text search.
Deep Dive: Why Models Fail
The paper includes a fascinating case study of a "Failed Search." In one instance, a model correctly identified a car brand (Changan) and its ambassador (Jay Chou), but when it reached the final step—identifying the color of a tie in a specific poster—it failed to "see" the image correctly. Instead of using its visual reasoning, it tried to infer the answer from text descriptions, leading to a "hallucinated" guess of blue when the ground truth was black.
Critical Analysis & Conclusion
VisBrowse-Bench exposes the "Last Mile" problem of multimodal agents: they are excellent at finding the relevant page, but mediocre at extracting the precise visual truth from it.
Takeaways for the Industry:
- Retrieval is not Reasoning: Simply adding an image search tool doesn't make an agent multimodal.
- Integration is Key: The massive gap between open-source (14%) and closed-source (47%) models suggests that fine-tuning for iterative tool-use logic is currently a massive competitive moat.
- Future Direction: We need models that can maintain "visual state" across multiple hops, essentially a "Visual Working Memory" for browsing.
As we push toward "Deep Research" agents, VisBrowse-Bench will likely become the gold standard for measuring whether an agent truly sees the web or is just reading the alt-text.