Can Semantic World Models Improve Mobile GUI Agent Task Success?

Why predicting GUI pixels is a poor target

A world model predicts what a system will look like after an action. The dominant recipe, borrowed from robotics and video generation, predicts future frames pixel by pixel. The paper argues this recipe breaks down on mobile screens. Using a news app scroll as its example, it lists what a pixel model would have to get right: understand that scrolling reveals more articles while most of the layout stays fixed, infer the layout of the new page, invent plausible news content, and render every text and icon. Even a model that did all four would rarely match the real next screen, because the real headlines at that moment are not predictable.

The paper's alternative is a semantic world model. Given the current screen and an action, it predicts a latent representation of the high-level changes, such as a search bar appearing or a new word entering an input box. That latent is queried in two ways: a text description of the expected change, and yes-or-no answers to statements about the next state. The authors formalize this by splitting the usual next-screen prediction into a semantic step and a separate rendering step, and they assume that once the changes are known, the next state can be determined without the action.

The motivating comparison in the paper is qualitative rather than a controlled benchmark: in its figure, a frontier image generator struggles to render the GUI state accurately, while a frontier vision-language model describes the expected changes in text. That example sets up the hypothesis the rest of the paper tests, not the test itself.

Two tasks that measure world modeling separately

MobileWorldBench evaluates a semantic world model with two tasks that answer different questions. Next-State-Generation asks the model to write a description of the expected state change from the current screen and action; a GPT-4o judge then scores that description against the real next screen on accuracy, completeness, and relevance, each on a 1-to-5 scale, and reports an overall score. Next-State-QA asks whether the model can judge statements about the next state, and is scored as answer accuracy. A model can be good at describing a transition yet weak at verifying specific claims, so the two scores should not be read as one capability.

The benchmark itself was built from human demonstrations. The pipeline converts low-level actions into high-level actions, annotates state-change descriptions, generates question-answer candidates, filters them with a vision-language model for self-consistency and relevance, and then has humans verify correctness and relevance. The resulting benchmark is fully human-verified, which matters because the training data described later is not.

The scale of the benchmark is modest by design: the paper reports 4,000 generated questions narrowing to 2,458 after model filtering, and 500 samples with 250 checked by humans, alongside 1,787 questions. These are evaluation items, not training examples, and the human check is what separates this benchmark from automatically filtered data.

Frontier models are weak next-state predictors

On Next-State-Generation without the ground-truth next image, the frontier models cluster tightly and low. Overall scores range from 11.81 for Intern-VL3-78B to 12.16 for Qwen3-VL-235B-A22B, with Gemini-2.5-Flash at 12.08, Claude-Sonnet-4.5 at 12.11, and GPT-4o at 12.00. The gap to the annotator models, which are given the real next image, is the informative part: those annotators reach 13.08 and 13.05 overall. The benchmark is therefore not saturated, and much of the headroom lies in knowing what actually happens next.

Fine-tuning on the authors' MobileWorld dataset moves the small Qwen3-VL-8B-Instruct model from 11.84 to 12.39 overall, a 4.7% relative gain. The breakdown shows where that gain comes from: accuracy rises 3.7%, completeness 8.2%, and relevance 2.7%, while the General category is unchanged at 12.83. The largest relative gains appear in System (+7.7%) and Web (+8.3%) categories, so the improvement is uneven across app types rather than uniform.

Next-State-QA tells a different story. Without the next image, humans answer 84.4% correctly on the full set, while the best frontier model, Gemini-2.5-Pro, reaches 79.1% and GPT-4o only 66.0%. Fine-tuning lifts Qwen3-VL-8B-Instruct from 67.3% to 71.4% on the full set and from 67.6% to 74.8% on the 500-sample TestMini subset. Even after fine-tuning, the model remains below human accuracy, and the two columns use different denominators, so the full-set and TestMini numbers should not be mixed.

What the MobileWorld dataset adds to a VLM

MobileWorld contains 1.4M samples drawn from human demonstrations in the AITW and Android Control training splits. Each sample pairs a current screen, an action, and the real next screen, with text descriptions and question-answer pairs generated around that transition. The final filtered dataset holds 942k state-change descriptions and 543k question-answer pairs. Annotations come from Qwen3-VL models rather than GPT-4o for cost reasons: 90% of the data is labeled by the 8B model and 10% by the 235B model, split into pretraining and fine-tuning portions.

Because this training data is not human-verified, the paper checks its pipeline on a subset and reports an error rate of 13.8% before automatic filtering and 4.7% after. That residual error is the main quality caveat for the dataset, and the authors note that more expensive filtering models could reduce it further. The benchmark and the dataset therefore carry different levels of assurance: one is fully human-verified, the other is automatically filtered and spot-checked.

Human evaluation adds a check that the automatic judge is not the whole story. In an arena-style comparison over 3,000 sampled matches, human judges preferred outputs from the fine-tuned model, which received the top ELO score of 925, above Gemini-2.5-Flash at 910 and Qwen3-VL-235B-A22B at 909. The paper also reports that the GPT-4o accuracy score predicted human preferences 83% of the time, which supports using that judge while leaving completeness less correlated with human preference.

World-model predictions raise task success rates

The payoff test is whether predicted state changes improve an agent's decisions. The paper uses a model-based policy: an action proposal model generates candidate actions, the semantic world model predicts the latent change for each, and a value model scores each predicted change against the high-level goal; the highest-scoring action is executed. The action proposal and value models are themselves vision-language models, so the world model is an added component rather than a replacement for the agent.

On AndroidWorld, Qwen3-VL-235B-A22B improves from 46.9% success to 49.1% with a value function for actions, to 50.8% with a zero-shot semantic world model, and to 54.3% with the fine-tuned world model. Gemini-2.5-Pro improves from 63.4% to 66.1% with the fine-tuned world model. These are success rates on the same benchmark, and the ordering shows that both the world model and its fine-tuning contribute, with the fine-tuned version adding more than the zero-shot one.

A zero-shot transfer test on WebVoyager moves a GPT-4o-based agent from 45.4% to 49.3% success. This is the paper's evidence that the benefit is not confined to the Android setting where the world model was trained, though the gain is smaller than the AndroidWorld improvement for the same kind of added component.

Where the gains are most reliable

The strongest evidence for semantic world modeling is the consistent direction of the agent results: adding the fine-tuned world model raises success on AndroidWorld for two different base models and on WebVoyager for a third agent. The weakest evidence is the absolute quality of the world model itself, since even fine-tuned models trail human accuracy on Next-State-QA and the benchmark's annotators outperform all frontier models when given the real next image.

The practical reading is that the value comes from ranking candidate actions, not from producing a faithful description of the future screen. The value model scores predicted changes against a goal, so a description that captures the relevant change can help even when it is incomplete. This also explains why the General category shows no gain after fine-tuning while System and Web categories improve: the benefit depends on whether the predicted change is decision-relevant for that app type.

The paper's own limitations point the same way. Generated outputs can hallucinate, and the training data's 4.7% residual error after filtering is not eliminated. Those caveats qualify the size of the reported gains rather than their direction, and they matter most for readers considering the dataset as a training resource rather than the benchmark as an evaluation tool.

The priority is the prediction target, not the renderer

The paper's central claim is a priority ordering: for mobile GUI agents, the useful part of a world model is the semantic prediction of what will change, and pixel-level rendering is a separate, harder problem that need not be solved first. The evidence for that ordering is that a fine-tuned 8B model improves next-state description scores and question answering, and that plugging its predictions into action selection raises AndroidWorld success from 46.9 to 54.3 and from 63.4 to 66.1, plus WebVoyager from 45.4 to 49.3.

The comparison that should guide a reader is between the zero-shot and fine-tuned world model on the same agent: 50.8 versus 54.3 on AndroidWorld. That gap says the dataset, not just the semantic formulation, carries much of the benefit. A reader deciding whether to adopt this approach should therefore weigh both the change in prediction target and the availability of in-domain transition data, since the paper's largest agent gains come from the fine-tuned model rather than the zero-shot one.

Sources used in this answer