How do multimodal models perform on interactive visual intelligence benchmarks?

A benchmark built from thousands of unfamiliar games

PlaySuite evaluates interactive visual intelligence: the ability to look at a changing screen, choose an action, and keep working toward a goal as the environment responds. The benchmark draws on games from PyWeek, which contributes lightweight Python titles, and itch.io, which adds games built in Unity, Godot, GameMaker, RPG Maker and other engines. The evaluated set contains 5,734 distinct games, drawn from a larger pool of roughly 17.5K launchable games that the authors' pipeline validated.

The scale matters for what the benchmark can claim. The paper states that PlaySuite covers over two orders of magnitude more game environments than prior multi-game benchmarks. Because the games are independently developed and largely out-of-distribution for current models, the authors argue that success is less likely to come from retrieving memorized walkthroughs or web-scale training artifacts. That is the paper's rationale for treating these games as a test of transfer rather than recall.

One control interface and one automated judge

Heterogeneous games create two practical problems: different engines expose different execution and control interfaces, and games do not share a common reward or scoring signal. PlaySuite addresses the first with a unified closed-loop interaction framework in which agents act through standard keyboard and mouse actions. The harness runs games in isolated containers with CPU, RAM and network limits and executes up to 768 concurrent games, which is what makes evaluation across thousands of titles feasible on an HPC cluster.

The second problem is solved by a Video-LLM-as-a-judge protocol. Instead of relying on per-game rewards, the judge extracts observable gameplay milestones from recorded gameplay and maps them to standardized progress levels. This produces a comparable progress measure across titles that were never designed to be scored the same way. The trade-off is built into the design: the metric captures observable progress at scale, not strategic efficiency, reasoning quality or mastery of underlying game mechanics.

Fourteen open models, three agent families

The evaluation covers fourteen recent open models spanning general-purpose vision-language models, computer-use agents and vision-language-action models. These families differ in how they turn perception into control: general VLMs ground instructions in pixels, computer-use agents predict control tokens for interfaces, and vision-language-action models are built to output actions directly. Including all three lets the paper ask whether specialization for interaction closes the gap.

The headline answer is that it does not close it. The paper reports that current models achieve meaningful gameplay progress on only a minority of games. Qualitative inspection of gameplay traces identifies recurring difficulties in spatial grounding, control selection, action execution, adaptation over time and passive waiting. These are failure categories from inspected traces, not a quantitative breakdown of error shares, so they should be read as the authors' characterization of what went wrong rather than as measured proportions.

Progress scores stay low across genres

The reported per-genre results use Mean Progress Score in the no-history condition at 0.33Hz, with genres included when they contain at least 60 games. On the itch.io corpus, the best overall model, Qwen3-Omni-30B-A3B, reaches scores such as 1.38 on simulation, 1.33 on strategy, 1.17 on shooter and 1.06 on adventure. Lower-ranked models fall well below that, with OS-Atlas-Pro-7B reaching 0.56 on adventure and 0.43 on puzzle.

The PyWeek results show the same pattern at smaller genre counts, and the paper explicitly cautions that per-genre counts there are small, so those numbers are primarily descriptive. This is an important boundary: the genre table supports a broad statement that progress is limited, but it does not support fine-grained claims that one model is reliably better than another within a specific genre.

More history and more prompting do not help

Two ablations test intuitive fixes. For temporal context, the authors vary history length across k = 0, 3, 5 and 10. Mean Progress Score on PyWeek falls from 0.946 with no history to 0.778 at k = 3, stays near 0.793 at k = 5 and drops to 0.740 at k = 10. The itch.io subset follows the same direction, from 0.616 at k = 0 to 0.511 at k = 10. The paper calls this counterintuitive: adding history does not improve average performance, though individual models can respond differently.

For prompting, the authors test an explicit strategy field and three few-shot examples on the eight VLMs that share a standard prompt template. Neither helps. The strategy field lowers mean progress on PyWeek from 0.785 to 0.747 while leaving itch.io nearly unchanged at 0.538 to 0.535, and few-shot examples reduce performance on both corpora, from 0.789 to 0.720 on PyWeek and from 0.538 to 0.504 on itch.io. The authors restrict this ablation to those eight VLMs because computer-use and vision-language-action models rely on model-specific formats and parsers, which would confound prompt effects with parsing failures.

Better progress comes at a steep compute cost

The cost analysis adds a comparison that raw scores hide. Qwen3-Omni improves mean progress over Qwen2.5-VL but requires 105.5 versus 59.9 H100 GPU-hours per 1,000 games. The paper attributes cost variation to model size, serving configuration and generation length. For readers comparing models, this means a higher Mean Progress Score is not automatically the better trade: the same benchmark can rank models differently depending on whether progress or compute per 1,000 games is the deciding constraint.

The paper also reports that broad model ordering stabilizes with a few hundred games, while fine-grained comparisons among the highest-ranked models need substantially larger samples. That is a practical threshold for anyone reusing the benchmark: a small subset is enough to separate strong from weak models, but not enough to resolve close calls at the top.

What the gap result does not cover

The evaluation is zero-shot, so it measures out-of-the-box generalization rather than maximum performance after fine-tuning or reinforcement learning. The closed-loop protocol also removes model-inference latency from game dynamics, so PlaySuite does not measure real-time reaction speed. And because the games, source code and metadata are publicly available, data contamination cannot be ruled out; adding newly released games can reduce but not eliminate that risk.

These limits shape how the central finding should be used. The perception-action gap is evidence about how current open models behave when dropped into unfamiliar interactive environments without adaptation, not a claim that interactive competence is unreachable. The failure categories the authors identify — spatial grounding, control selection, action execution, adaptation and passive waiting — are the specific targets that this benchmark is designed to track over time.

Where the benchmark changes the comparison

The most decision-relevant shift is the unit of comparison. Static visual reasoning scores and computer-use results do not predict gameplay progress here: models that look strong on those tasks still reach meaningful progress on only a minority of the 5,734 games. For a reader choosing among open models for interactive work, the paper's evidence favors checking sustained progress on unfamiliar environments and the compute cost per 1,000 games together, rather than extrapolating from static benchmark rankings.

The second shift is about what to fix. Because added history and added prompt structure both failed to raise average progress, the paper's evidence points away from simply giving models more context or more explicit instructions. It points toward the failure modes observed in the traces: grounding actions in space, selecting the right control, executing it, and correcting after mistakes. Those are the capabilities PlaySuite is built to measure as new models appear.

Sources used in this answer