Which Visual Representations Do World Action Models Need for Manipulation?

Inherited VAE features are built for reconstruction

World Action Models (WAMs) adapt pretrained video generation models to robot control by jointly predicting future video latents and action chunks. That inheritance brings a strong visual prior, but it also brings the latent space of the video model's variational autoencoder (VAE), which was optimized to reconstruct images rather than to preserve the information manipulation needs, such as object semantics and spatial relationships.

The paper's starting measurement shows the cost of that mismatch. On LIBERO-Plus, a benchmark that perturbs observations across seven categories, the VAE-only WAM averaged 74.7% success. Its weak points were viewpoint changes (51.2%), background changes (62.0%), and image noise (66.6%), while language perturbations (91.1%) and lighting changes (94.3%) were already high. The question is therefore not whether external visual features help, but which representations and which integration mechanism deserve the effort.

Earlier work often paired a chosen representation with a particular integration design, so the effect of the representation and the effect of the mechanism could not be separated. This study fixes the WAM backbone, training data, observation inputs, action horizon, optimizer, and learning-rate schedule, then varies one factor at a time: thirteen frozen visual encoders, five integration mechanisms, and combinations of sources.

General-purpose features beat capability-specific encoders

All thirteen encoders were frozen and added to the VAE-only baseline through channel-wise concatenation, and each was scored on all seven LIBERO-Plus perturbation categories across three evaluation seeds. The paper groups them by pretraining emphasis: general-purpose sources intended for broad transfer, and specialized sources trained for one capability such as segmentation, geometry, or interaction.

General-purpose representations averaged 82.2% success against 80.0% for specialized ones. DINOv3, a self-distillation encoder, reached 84.0% overall and improved the categories where the baseline was weakest, including camera (64.2%), background (83.9%), and noise (86.0%). V-JEPA 2.1 followed at 82.4%, and C-RADIOv4 at 83.5%. Specialized encoders did not close the same gaps: SAM 3 and VGGT-Ω each reached 80.6%, and the interaction-oriented HRP reached 79.8%, below the 74.7% baseline only in the sense of a modest gain.

The authors' reading is that a specialized encoder chosen to patch an assumed missing capability does not reliably deliver that capability under perturbations relevant to it, whereas a general-purpose representation provides broader robustness. The practical consequence is a default: begin with a strong general-purpose source rather than reasoning backward from a diagnosed weakness.

Channel-wise joint prediction gives the largest average gain

The five mechanisms differ in how external features enter the model. REPA aligns intermediate WAM hidden states to external features with a cosine loss. Gated cross-attention lets action hidden states query projected current-observation features through a learnable gate. Channel-wise concatenation spatially aligns features to the VAE latent grid and concatenates them along channels, expanding the video expert's projections so it predicts both VAE and feature flow velocities. Token-wise concatenation keeps features as a separate token sequence sharing the video expert's Transformer layers. A feature expert instead gives features their own Transformer parameters.

Averaged over the thirteen representations, channel-wise concatenation reached 81.5%, a 6.8-point gain over the 74.7% VAE-only baseline. REPA (76.2%) and the feature expert (76.4%) produced small gains, gated cross-attention was unchanged at 74.7%, and token-wise concatenation fell to 72.7%.

Two follow-up analyses support the authors' explanation of why channel-wise concatenation works. Within that mechanism, predicting only current features gave 76.5%, predicting only future features gave 81.9%, and predicting both gave 84.0%, so future-feature prediction carries most of the benefit. A sensitivity analysis of final normalized actions to current DINOv3 features also indicated that the action expert uses the added visual information more effectively under this mechanism. These are the authors' mechanistic interpretations of measured ablations, not separately proven causal claims.

Extra sources help only when they add something new

Adding a second representation does not reliably add success. DINOv3 alone improved LIBERO-Plus by 9.3 points over the VAE-only baseline, while the best evaluated combination added at most a further 3.7 points. The paper's explanation is overlap: sources trained on similar objectives or with shared teachers contribute redundant information.

The clearest case is C-RADIOv4, which distills DINOv3, SigLIP 2, and SAM 3; combining it with DINOv3 reduced LIBERO-Plus success to 78.6%. SigLIP 2 improved DINOv3 by 1.8 points, but reduced success by 0.4 points once UniTok was present and added only 0.2 points when V-JEPA 2.1 was also included. VGGT-Ω and SAM 3 changed little when added to DINOv3, although both improved some larger combinations. By contrast, adding V-JEPA 2.1 to DINOv3 and UniTok helped, consistent with its video-based predictive pretraining supplying complementary cues.

The resulting rule is about information, not count: select an additional source for the task-relevant information it contributes beyond what is already present, and treat composition as a refinement rather than the main lever. The paper states this directly, and the numbers show why, since the first general-purpose addition is worth more than any evaluated composition step.

The recipe transfers to RoboCasa and RoboTwin

The authors instantiate the findings as R2-WAM: a strong general-purpose representation, channel-wise joint prediction, and selectively retained additional sources. They then test it beyond the controlled LIBERO-Plus comparisons, where the VAE-only baseline is 74.7%.

On RoboCasa GR1 Tabletop, R2-WAM reaches 84.7% success across 24 tasks with 50 trials per task, 5.9 points above the VAE-only baseline. On RoboTwin 2.0 it reaches 92.8% across 50 tasks with 100 trials per task in each of the clean and randomized settings, 2.2 points above the corresponding baseline. The paper also reports validation across different WAM formulations on LIBERO-Plus and on four real-robot tasks with 20 trials per task per model.

These are different benchmarks with different task counts, trial counts, and baselines, so the percentage-point gains should not be compared as if they measured the same difficulty. What they jointly support is narrower and more useful: the recipe's advantage over a VAE-only model persists when the backbone formulation and the evaluation environment change.

The first representation choice matters most

The study's own comparison sets the priority. Moving from the VAE-only baseline to a strong general-purpose representation produced a 9.3-point LIBERO-Plus gain, while the best evaluated composition added no more than 3.7 further points. Integration mechanism mattered too, but only channel-wise concatenation produced a large average gain; the other four mechanisms ranged from a 2.0-point loss to a 1.7-point gain.

The authors therefore direct future work toward stronger general-purpose visual representations and more effective mechanisms for integrating them into WAMs, with feature composition as a supplementary route. That ordering is the paper's most decision-relevant output: if a WAM still relies on its inherited VAE latents, the largest available improvement lies in the representation and integration choice, not in stacking more feature sources.

The scope is bounded by what was tested. The controlled comparisons use frozen encoders, a Wan 2.2-initialized video expert with a separate action expert, and LIBERO-Plus as the main robustness benchmark; the transfer results cover RoboCasa GR1 Tabletop, RoboTwin 2.0, and four real-robot tasks. The composition findings in particular are drawn from the evaluated combinations, so the overlap-based reasoning is guidance for selecting sources rather than a guarantee about untested pairs.

Sources used in this answer