cotomi Act: Transforming Browser Agents from Stateless Executors to Situated Co-workers

TL;DR

cotomi Act is a next-generation browser agent that doesn't just execute commands—it learns your job by watching you. By combining a high-efficiency Agent Scaffold (reaching 80.4% on WebArena) with a Behavior-to-Knowledge pipeline, it converts raw UI interactions into a shared organizational wiki. This allows the agent to navigate internal jargon and tacit workflows that traditional LLM agents typically fail to grasp.

Background: The Gap Between Reasoning and Execution

In the world of Computer-Using Agents (CUAs), we have seen a surge in reasoning capabilities. However, even the best models often operate in a vacuum. They don't know who your manager is, which "Submit" button matters in a messy internal ERP, or what your specific "approval workflow" looks like.

The authors identify two fatal flaws in existing systems:

1. Context Bloat: Including every DOM element at every step kills the model's reasoning window.
2. Lack of Tacit Knowledge: RAG on static company docs isn't enough; the most valuable info—how people actually work—is never written down.

Methodology: The Two Pillars of cotomi Act

1. The Efficient Scaffold

To beat the human baseline on WebArena, cotomi Act introduces several performance-pumping mechanisms:

• Adaptive Lazy Observation: Instead of sending the whole page, it sends viewport-visible elements and only fetches the full tree on demand, reducing latency by 2.6x.
• Verbal-Diff History: Instead of a full history of page states, it uses natural language descriptions of what changed (e.g., "New element: Submit button appeared"). This reduces token cost by 3.8x.
• Test-Time Scaling (TTS): Uses best-of-N action selection to filter out hallucinations before they happen.

Figure 1: The architecture showing the interplay between the Behavior Logger, the Knowledge Workspace, and the Agent Scaffold.

2. Behavior-to-Knowledge Pipeline

This is the "secret sauce." The system passively monitors your browsing. An agentic ETL (Extract, Transform, Load) pipeline segments these logs into "task episodes" and distills them into:

• Activity Timelines: What was done and when.
• Task Boards: Proposing status updates (e.g., "I saw you finished this report, shall I mark it Complete?").
• Wiki Pages: Procedural scripts distilled from your actions.

Experiments & Results: Surpassing the Human Baseline

The results on WebArena are striking. By optimizing the scaffold, cotomi Act achieved 80.4% success, crossing the 78.2% threshold of human performance—a milestone in web navigation agents.

Table 1: cotomi Act vs. SOTA systems. Note the "fmt" column indicates the agent requires no site-specific hints.

Furthermore, the paper proves that knowledge matters. On the WorkArena benchmark (representing enterprise tasks), the success rate jumped significantly as more "observed behavior" was added to the agent's wiki. In some categories, success went from 75% to nearly 100%.

Figure 2: The more "Behavioral Knowledge" the agent has, the higher its success rate across different domains.

Critical Analysis: Privacy and Curation

The strength of cotomi Act is its Shared Knowledge Workspace. By externalizing the agent's memory into a human-readable format, it solves the "black box" problem. If the agent learns a "wrong" workflow, the user can simply edit the wiki.

Limitations:

• The behavior logger is currently "opt-in" for privacy, but in a real-world enterprise, the noise of "interrupted tasks" or "mistakes" by the user might contaminate the knowledge base.
• Test-time scaling adds latency, which may be a hurdle for real-time interaction.

Conclusion

cotomi Act represents a shift from agents that "know how to code/reason" to agents that "know how YOU work." By turning observation into structured organizational knowledge, it bridges the gap between a generic LLM and a productive digital colleague.