How could web-task harness evaluation change browser agents over the next two years?

Security will become a core evaluation metric, not an afterthought

Today's web-task harnesses mostly test whether an agent can complete a task, not whether it can be tricked into doing something harmful. But a 2026 threat model from Datta et al. shows that browser agents are vulnerable to 20 distinct attacks, including 10 classic web social engineering tricks that reemerge in amplified forms when an agent is influenced by untrusted page content [1]. This means that a harness that only checks task success will miss critical failures where an agent is manipulated into clicking a malicious link or leaking data.

The same study found that these attacks reproduce across four major LLM vendors, so this isn't a one-model quirk [1]. Over the next two years, expect harnesses to include adversarial scenarios—like pages with hidden instructions or deceptive UI—as standard practice. This will force agent designers to build in defenses, such as verifying actions against a user's intent, before agents are trusted with real-world tasks.

Harnesses will push agents beyond simple browsing to handle the chores people avoid

Most current benchmarks focus on general browsing, but real users need agents to handle tedious, multi-step tasks like filling out forms or comparing prices across many pages. WebChoreArena, a 2025 benchmark with 532 tasks, specifically tests these 'chores' by adding three challenges: massive memory (retrieving lots of info), precise calculation, and long-term memory across pages [2]. Even the best model tested, Gemini 2.5 Pro, still showed 'substantial room for improvement' compared to the simpler WebArena benchmark, meaning these tasks are genuinely harder [2].

This matters because harness evaluation will increasingly use such benchmarks to measure progress. As models improve on WebChoreArena, we'll see agents that can reliably handle the tedious work people hate—but the gap also shows that current agents aren't ready for prime time on complex tasks. Harness designers will need to add features like better memory management and step-by-step verification to close this gap.

The harness itself—not just the model—will be the key differentiator

A 2026 position paper argues that an agent's reliability depends on the 'harness layer'—the code that controls what actions are allowed, how state is managed, and how failures are handled—not just the underlying LLM [3]. This is a shift from thinking that a better model automatically means a better agent. The paper proposes a framework (Control, Agency, Runtime) and suggests that many reported agent gains may be 'harness-sensitive,' meaning the same model can perform differently depending on how the harness is built [3].

For users, this means that over the next two years, we'll see more focus on harness engineering—like adding safety checks, memory systems, and error recovery—as a way to improve agent performance. Evaluation will need to report not just the model but also the harness design, so that comparisons are fair and reproducible. This is good news: it means progress can come from better engineering, not just waiting for the next model release.

About These Sources

This answer is built on 3 studies (all preprints) — published from 2025 to 2026, 3 from 2024 or later — selected as the most relevant from 3 studies that passed quality screening, drawn from 29 papers retrieved from a database of over 500 million.

Sources used in this answer

Related research questions