[Preprint 2025] Gym-Anything: Scaling Computer-Use Agents via GDP-Grounded Environment Synthesis

TL;DR

The bottleneck for AI agents isn't just the model; it's the environment. Gym-Anything automates the creation of complex software environments (from Photoshop to medical software) using an agentic "Creation-Audit" loop. The resulting CUA-World benchmark features 10,000+ tasks grounded in real-world U.S. GDP data, revealing that current frontier models are still far from mastering long-horizon professional workflows.

The "Wallpaper" Problem: Why Current Agents Lack Economic Value

If you look at current CUA benchmarks like OSWorld or Mind2Web, agents are mostly tested on "toy" tasks: opening a browser, changing a system setting, or filling a simple form.

The Gap: Real professional work is messy. It happens in specialized software (CAD, EMR, ERP) and requires hundreds of steps. Why haven't we benchmarked this? Because setting up a "Radiology environment" with real CT scans and patient records takes a human expert weeks. We are stuck in a "low-complexity" loop because our evaluation environments don't scale.

Methodology: Let Agents Build the Gym

Gym-Anything treats environment creation as a coding + computer-use task. Instead of manual VM snapshots, it defines an environment via a "Specification Schema" (Install, Configure, Task-Setup scripts).

1. The Creation-Audit Loop

To ensure quality, the framework employs an adversarial multi-agent setup:

• Creation Agent (AgentC): Researches the software, writes bash/python scripts, and downloads real datasets (e.g., USGS earthquake data).
• Audit Agent (AgentAudit): Distrusts the Creator. It ignores comments and inspects "evidence" (screenshots and logs) to ensure the software isn't just stuck on a login screen or using placeholder data.

2. GDP-Grounded Selection

The authors didn't just pick "popular" apps. They mapped 894 U.S. occupations to the software they use and attributed GDP value to specific applications. This ensures the benchmark measures what actually moves the economy.

CUA-World-Long: The New Frontier of Failure

The authors introduced CUA-World-Long, a subset of tasks requiring over 200 (and up to 500+) steps. This is where even the "God-tier" models fail.

Why do they fail?

• The Scrolling Loop: Analysis shows agents get stuck in "UI exploration"—scrolling endlessly through medical image slices or menus without ever "committing" to an action.
• Premature Termination: Agents often claim a task is done when they've only completed 20% of the workflow.

Scaling Results: Distillation and Auditing

The paper provides two massive "Aha!" moments for the industry:

1. Distillation Works: By training a tiny 2B student model (Qwen3-VL) on successful trajectories from a strong teacher, they created a model that outperforms 4B models. The performance follows a log-linear scaling trend: doubling the training data (software and task varieties) yields a consistent ~3.5 point gain.
2. Test-Time Auditing (TTA): By running a separate VLM to "audit" the agent's work at the end of a trajectory and providing feedback (e.g., "You forgot to save the report"), performance on long-horizon tasks jumped significantly.

Deep Insight: Visual Complexity is the "Floor"

A fascinating finding in the ablation studies was the impact of Visual Complexity. Large models (Gemini-3, Kimi) are robust across visual styles. However, for smaller models (2B-4B), "High Complexity" UIs (like 3D viewports in Blender or Slicer3D) act as a complete performance floor—distillation helps, but the "visual gap" remains steep.

Conclusion

Gym-Anything moves us away from static, hand-crafted benchmarks toward a world of dynamic, economically-relevant evaluation. It proves that the path to SOTA agents isn't just more parameters, but more diverse, high-fidelity interaction data in environments that finally look like a real office.

Main takeaway: If you want to build a better agent, use an agent to build a better gym.