[CVPR 2024] GUI-CEval: Stress-Testing the Limits of Chinese Mobile GUI Agents

TL;DR

GUI-CEval is the first large-scale, hierarchical benchmark tailored for the Chinese mobile ecosystem. Spanning 201 apps and 4 device types, it moves beyond simple "click-here" grounding to evaluate the entire cognitive chain: Perception, Planning, Reflection, Execution, and Evaluation. The verdict? While visual perception is nearing maturity, current SOTA models like Qwen2.5-VL and UI-TARS still face a "performance cliff" in multi-step, real-world interactions.

Background: The Linguistic and Ecosystem Gap

While the AI world has seen a surge in GUI agents (e.g., AndroidWorld, ScreenSpot), these benchmarks suffer from a heavy English-centric bias. The Chinese mobile ecosystem—with its unique UI layouts, "super-apps" (like WeChat/Alipay), and distinct interaction patterns—remained a blind spot. Moreover, previous benchmarks often evaluated perception (What do I see?) and action (What do I do?) in isolation, making it impossible to diagnose why an agent failed in a real-world task.

Methodology: A Hierarchical Diagnostic Framework

GUI-CEval breaks the "black box" of agent behavior by splitting evaluation into two distinct tiers:

1. Foundation Tasks (The Atomic Skills): Using a unified multimodal QA format, the benchmark tests 11 sub-tasks across five dimensions. It uses Set-of-Marks (SoM) to reduce coordinate sensitivity, forcing the model to choose regions rather than guessing pixels.
2. Application Tasks (The Real-World Test): This is where the rubber meets the road. Agents are tested on Online Interaction—facing real network lag, pop-up ads, and permission prompts on physical devices.

Figure 1: The GUI-CEval pipeline, integrating human-verified trajectories with atomic skill diagnostics.

Key Insights: Why do Agents Fail?

The authors evaluated 20 representative models, including GPT-4o and the Qwen series. Several "Hard Truths" emerged:

1. The "Reflection" Bottleneck

While models score high in Perception (up to 82%), they collapse in Reflection (max 21.01%). Most agents cannot recognize when they have made a mistake or landed on an error page, leading to "looping" behavior where the agent repeats the same incorrect action indefinitely.

2. The Performance Cliff

There is a monotonic decline in success rate as the number of steps increases. For tasks requiring 3 steps, top models manage a ~50% success rate. However, for tasks longer than 7 steps, success rates drop to nearly zero. This "cliff-like" decline suggests that error accumulation is the primary enemy of long-horizon agents.

Figure 2: Performance radar charts showing the imbalance between perception and higher-order reasoning.

3. Scaling vs. Domain Specialization

Larger models (e.g., Qwen2.5-VL-72B) generally perform better, but domain-specific SFT (Supervised Fine-Tuning) allows smaller models (like UI-TARS-2B) to punch above their weight, especially in grounding and offline execution.

Critical Analysis: The Road to "Super Agents"

GUI-CEval provides the most sobering look yet at the gap between "demo-ready" and "deploy-ready" agents.

• Resolution Sensitivity: The research finds that shrinking screenshots to 1/2 size maintains performance, but 1/6 size causes a total collapse. This has massive implications for on-device inference, where memory and bandwidth are at a premium.
• The Initial State Problem: Most benchmarks start the agent on a clean "Home" screen. GUI-CEval proves that if you start an agent mid-way through an app or on a random page, success rates plummet. Policy transfer and state awareness remain unsolved.

Figure 3: Correlating offline Step Success Rate (SSR) with online success—note the diminishing returns.

Conclusion

GUI-CEval is more than just a leaderboard; it is a diagnostic map. It tells us that making an agent "see" is no longer the challenge. The real battle lies in Process Supervision—teaching agents to evaluate their own progress and recover from the chaotic unpredictability of a real mobile OS. For developers in the Chinese AI space, this is now the definitive yardstick for progress.