[CVPR 2026] MolmoWeb: Building the SOTA Open Visual Web Agent from Screenshots

Executive Summary

TL;DR: MolmoWeb is a new family of fully open-source multimodal web agents (4B and 8B parameters) that navigate the internet just like humans do—by looking at screenshots. By eschewing brittle HTML/DOM parsing in favor of a pure visual-language policy, and training on the massive MolmoWebMix dataset, these models outperform proprietary giants like GPT-4o on standard web-use benchmarks.

Background: Within the "computer-use" paradigm, most agents rely on structured data (AxTree) which is often messy and computationally expensive. MolmoWeb represents a significant shift toward vision-centric autonomy, proving that smaller, well-trained models can dominate the leaderboard if they possess superior spatial grounding and reasoning capabilities.

Problem & Motivation: The Brittle Web

Most existing web agents are "blind" in the sense that they read the underlying code (HTML/AxTree) rather than seeing the page. While this provides text-rich data, it suffers from several fatal flaws:

1. Complexity: AxTree inputs can consume tens of thousands of tokens per page, leading to high latency and cost.
2. Brittleness: DOM structures vary wildly across frameworks and break whenever a site is updated.
3. Incompleteness: Dynamically rendered content often doesn't show up correctly in the Accessibility Tree.

The authors argue that a truly robust agent must use the same perceptual modality as humans: Vision.

Methodology: MolmoWebMix and The Visual Policy

The core innovation lies in MolmoWebMix, a diverse data cocktail designed to teach a model not just how to click, but why.

1. Data Composition

The dataset includes four distinct pillars:

• Human Demonstrations: High-quality trajectories with natural language "thoughts."
• Synthetic Trajectories: Generated by larger "teacher" models (like Gemini-3-Flash) that do have access to the AxTree, providing a "gold standard" for the student to imitate visually.
• Atomic Skills: Focused training on specific sub-tasks like "form filling" or "product filtering."
• GUI Perception: Over 7 million QA pairs to sharpen the model's OCR and element localization (grounding).

2. Model Architecture

Built on the Molmo2 architecture, the model functions as an action policy: $ext{Action}_t=f(ext{Screenshot}_t,extInstruction,ext{History}_{t-10})$

Figure 1: The MolmoWeb execution loop. The agent predicts a "Thought" (Chain of Action) before outputting a JSON-formatted browser action.

Experiments & Results

MolmoWeb establishes a new SOTA for open-weight models, but more impressively, it beats closed-model baselines that use more complex inputs.

Performance Highlights

• WebVoyager: MolmoWeb-8B reached 78.2% pass rate, surpassing OpenAI’s Computer-Use-Preview (70.9%).
• Test-Time Scaling: By running 4 parallel attempts and using a VLM to judge the best one (Pass@4), the 8B model soared to 94.7% on WebVoyager.

Table 1: MolmoWeb consistently outperforms comparable 7B-9B models and competes with GPT-5/o3 class models.

The "Synthetic > Human" Paradox

In a fascinating ablation, researchers found that training on synthetic trajectories actually yielded better results than training on human ones. Why? Humans often take "noisy" detours or exploratory clicks, whereas LLM-generated trajectories (using AxTree grounding) are direct and provide a cleaner signal for the model to learn spatial coordinates.

Critical Analysis & Conclusion

Takeaway: MolmoWeb proves that specialized, vision-only visual models are the future of digital agents. They are faster, cheaper, and arguably more robust than their text-heavy predecessors.

Limitations:

• Dynamic Latency: Though the model is efficient, the bottleneck remains the time it takes for a live browser to render pages.
• Exploration: The model still struggles with vague instructions that require deep multi-page exploration without a clear URL.

Future Work: The release of the full MolmoWebMix dataset is a gift to the research community. It paves the way for advanced self-distillation and Reinforcement Learning (RL) techniques that could push the Single-Rollout performance even higher.