WebCompass: Beyond HumanEval — Evaluating the Full Lifecycle of Web Engineering
WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
WebCompass is a comprehensive multimodal benchmark designed to evaluate Code LLMs across the full lifecycle of web engineering, covering generation, editing, and repair tasks. It introduces a novel "Agent-as-a-Judge" protocol that utilizes the Model Context Protocol (MCP) to interact with generated websites in real browsers, achieving a high human-agreement correlation (Pearson r > 0.93).
TL;DR
WebCompass is a massive leap forward in how we measure "Web Intelligence." Moving beyond simple text-to-code snippets, it introduces a 1,526-task multimodal benchmark that tests if AI can Generate, Edit, and Repair complex websites. Using a sophisticated Agent-as-a-Judge system, it evaluates models by actually "clicking and browsing" their output, revealing that even top-tier models like Claude-4.5 and Gemini-3-Pro still struggle with visual aesthetics and complex framework logic.
The Blind Spot in Modern Code Benchmarking
Most current benchmarks (like HumanEval or MBPP) treat coding as a localized logic puzzle. But web development is different—it's visual, interactive, and iterative. The authors argue that a model scoring perfectly on functional logic might still produce a website that is:
- Visually Broken: Overlapping text, poor contrast, or "white screens."
- Interaction-Dead: Buttons that look clickable but lack event listeners.
- Non-Maintainable: Fails when asked to "edit" a feature inside an existing 32k-token codebase.
WebCompass addresses this by treating the LLM as a Web Engineer, not just a coder.
Methodology: The "Agent-as-a-Judge" Revolution
The core innovation of WebCompass is its evaluation strategy. For open-ended generation (where multiple "correct" designs exist), it discards static string matching for Dynamic Interaction.
The Evaluation Pipeline
- Checklist Generation: An LLM-PM creates a design spec.
- Autonomous Exploration: Using the Model Context Protocol (MCP), a judge agent (Claude Code) launches the generated site in a headless Chromium browser.
- Adaptive Verification: The agent writes temporary JS test cases to probe the DOM and CSS, adapting to the model's specific naming conventions while maintaining the behavioral "Ground Truth."
Figure: The WebCompass Lifecycle — spanning Generation, Editing, and Repair across Text, Vision, and Video.
Key Insights from the Experiments
1. The "Aesthetics Gap" is the New Frontier
While top models excel at Runnability (getting the page to show up), they fail at Design Quality. Even the best-performing models show a significant drop when moving from functional implementation to visual polish. Aesthetics remain the most persistent bottleneck, specifically for open-source models.
2. Framework Sensitivity: The Vue Problem
One of the most fascinating findings is how framework choice affects model "IQ."
- Vanilla JS is the easiest for models to generate/edit because there is no build-tool "magic."
- Vue is consistently the most difficult. The authors hypothesize that Vue’s Single-File Component (SFC) structure—which mixes HTML, Scoped CSS, and JS logic in one block—creates "attention dilution," leading to more cross-block inconsistencies.
Figure: Radar chart showing the massive gap between closed-source giants (Claude/Gemini) and open-source models (Qwen3) across the seven task dimensions.
3. Repairing vs. Generating
Repairing a bug is fundamentally harder for models than generating code from scratch. The data shows that Root-Cause Targeting (finding the actual bug) is the lowest-scoring dimension. Models often "over-edit," fixing the visual symptom but introducing new bugs in the surrounding logic.
Critical Analysis & Future Outlook
WebCompass proves that we are entering the era of Agentic Evaluation. We can no longer rely on static files to judge AI quality.
Limitations: However, the benchmark is currently front-end only. A true "Web Engineering" benchmark would need to include DB schema design and API consistency (Back-end). Additionally, the cost of running "Agent-as-a-Judge" (launching browsers for every test) is high, potentially limiting its use for smaller research teams.
Conclusion: WebCompass sets a new standard. If you want to build a "Devin-killer," it’s no longer enough to pass LeetCode. You have to prove your agent can navigate a React repo, fix a z-index occlusion, and maintain visual harmony across a multi-page app.
Table: WebCompass vs. Prior Benchmarks. Notice its unique support for Video-input and Agentic testing.