Can Transferable Adversarial Patches Attack World Action Models Without Target Queries?

An attack that never queries the victim policy

Most prior attacks on world action models optimize against the victim's actions or predicted futures, which means the attacker needs access to target-model outputs. TAPDreamer removes that requirement. It optimizes a patch against a public visual encoder alone, using six demonstration frames drawn from one source task, and then freezes the patch pixels, area, and position for the rest of the evaluation.

The optimization objective is deliberately interface-agnostic: maximize the global L1 distance between the encoder's clean and patched final latents, averaged over the J source frames and normalized by the total number of scalar elements in that latent. The paper reports J = 6, 500 Adam gradient-ascent updates, a learning rate of 0.02, and pixel initialization uniform in [-1, 1] with projection back to that range after each update. Because the search never touches action heads, rewards, or simulator rollouts, the same artifact can be deployed unchanged across tasks and across different action architectures within a benchmark.

What 6.5% coverage actually buys

On LIBERO the external and wrist views form a 224x448 canvas, and the main patch occupies an 81x81 region in the upper-left corner of the external view, covering 6.54% of the input. The corresponding coverage on RoboTwin is 6.59%. These are the numbers behind the roughly 6.5% figure, and they matter because the comparison baseline is a random patch matched to the adversarial patch in both position and area.

That matched baseline is what separates the result from ordinary occlusion. On FastWAM, the clean success rate is 97.65% on LIBERO and 90.86% on RoboTwin. Random patches retain 81.45% and 79.20% respectively, so local pixel replacement alone costs 16.20 and 11.66 percentage points. TAPDreamer takes both to 0.00%. The gap between 81.45% and 0.00% is the part attributable to optimization rather than to covering part of the image.

How a corner patch reaches the whole latent

The paper's central mechanism claim is that the patch does not stay local. The VAE encoder's bottleneck contains spatial self-attention on a 14x28 grid, and the authors measure the fraction of squared representation change falling outside the patch footprint, defined as the projection of the patch region onto that feature grid. For TAPDreamer this fraction rises from less than 1% at the attention input to about 71% at the attention output and 79% in the final latent. A random patch of the same size leaves changes largely inside the footprint.

Decomposing the attention output as H = U + AVW_O, the paper isolates three terms: changed values, changed attention weights, and their joint change. The joint term is the largest measured component for TAPDreamer but small for a random patch, which the authors read as coordinated changes in weights and values rather than simply more attention paid to the patch. Restoration tests support that reading: resetting the attention branch, the weights, or the values to their clean counterparts while keeping the patched residual branch substantially reduces latent and action changes, and suppresses perturbations outside the footprint. The same interventions barely affect the random patch.

One source template describes held-out scenes

Transfer across tasks requires more than spatial spread within a single observation; the response has to repeat when the scene changes. The paper averages patch-induced latent changes over observations from 12 source-task states to form a fixed template, then compares it against changes on 36 held-out states, restricting statistics to latent positions outside the projected patch footprint.

The source template reaches a median cosine similarity of 0.981 with held-out changes, and the median relative L1 residual is 0.206. Clean representations from the same observations have a cross-task cosine similarity of 0.371, so the shared direction is specific to the patch-induced change rather than a property of the scenes. Magnitude also survives: using the measured residual per held-out state, the median ratio of the lower bound in the reverse-triangle-inequality decomposition to the observed L1 displacement is 0.833, with a range of 0.732 to 0.918. About 95.3% of final-latent change energy outside the footprint lies in a shared spatial component whose direction has a cross-task cosine similarity of about 0.997.

Transfer holds across action architectures

The headline numbers come from FastWAM, but the paper also reports transfer within the same benchmark to other configurations. The same patches reduce success to 2.1% and 0.8% on two DreamWAM configurations and to 10.0% on Motus. Each model configuration retains its own action generation process and preprocessing, and cross-model transfer occurs within each benchmark rather than across benchmarks.

These are different denominators from the FastWAM results and should be read separately: 2.1%, 0.8%, and 10.0% are not directly comparable to the 0.00% figures on 40 LIBERO and 50 RoboTwin tasks. What they establish is that a patch optimized against a public encoder, with no queries to any of these policies, still degrades architectures it was never optimized against.

Where the conclusion stops

All reported evaluation is closed-loop simulation on LIBERO and RoboTwin, with attacks implemented by modifying model input images. The paper does not report physical-robot trials, and the patches remain visible throughout each trial in standard attack evaluations, so the results describe a sustained, unoccluded local perturbation rather than a brief or intermittent one. The optimization frames come from public LeRobot demonstrations, filtered by source-task identifier, with one frame selected from each of six episodes at temporal indices 5, 20, 40, 60, 15, and 35.

The defense discussion is explicitly forward-looking. The authors suggest attenuating keys and values from suspicious image regions or training the encoder to preserve representations outside locally perturbed regions, but state that effectiveness against adaptive attacks and impact on clean-task performance remain to be evaluated. Those proposals follow from the attention-mediated spread the paper measures; they are not validated defenses reported here.

Why encoder-level defense is the priority

The paper's own framing is that protecting downstream action generation alone is insufficient. The evidence for that priority is the combination of two measurements: a patch optimized only against a public encoder transfers to FastWAM, two DreamWAM configurations, and Motus without any target-policy queries, and the mechanism analysis locates the damage in attention-mediated spread from a local region to the full latent. If the vulnerability lives in the shared visual representation rather than in any single action head, then defenses scoped to one policy or one action interface leave the same channel open.

For readers weighing where to spend effort, the paper's most decision-relevant split is between the matched random patch and the optimized one: 81.45% versus 0.00% on LIBERO and 79.20% versus 0.00% on RoboTwin. That gap, not the raw 0.00%, is what distinguishes an optimized persistent local perturbation from ordinary occlusion, and it is the comparison any defense evaluation for world action models should be required to close.

Sources used in this answer