How can simulation motion priors be distilled into real-domain robot world models?

SimForcing transfers motion, not simulated appearance

Action-conditioned robot world models have to respond precisely to continuous robot trajectories while still producing realistic, temporally coherent visual dynamics. Learning both from heterogeneous real robot videos is hard, and simulation offers structured motion supervision that real videos lack. The obstacle is that simulated and real observations look different, so directly matching them is unreliable, and predicted simulation trajectories are themselves imperfect, so a real-domain model that leans on them too heavily can inherit their errors.

SimForcing separates these two problems. It uses simulation as a source of transferable motion knowledge and as a controllable reference for prediction. The first role is handled by latent-motion distillation: rather than matching appearances, the method aligns temporal changes between adjacent teacher and student video latents, so the student internalizes simulation-derived motion priors while appearance differences are largely bypassed. The second role is handled by multi-block simulation conditioning with condition dropout, which exposes the student to predicted simulation latents during training but keeps it able to predict without them.

A simulation teacher initializes the student

The pipeline starts with a mixture-of-transformers world model trained on synthetic rollouts with trajectory augmentation, learning action-dependent motion from diverse robot trajectories. That model then initializes the student and serves as a frozen teacher. The video backbone is based on Wan2.2-TI2V-5B, with 30 transformer blocks, a hidden dimension of 3,072 and 24 attention heads; the action transformer also has 30 blocks with a hidden dimension of 1,024.

Motion supervision matches adjacent-frame differences in the estimated clean video latents and is applied to every training sample. The real-world prediction branch additionally receives paired simulation latents with probability 0.7, injected through residual projections initialized to zero, while the simulation prediction branch does not receive this auxiliary condition. Initial-frame and action conditions are retained throughout training, and the strength of motion supervision is unchanged when the auxiliary simulation condition is present. Training jointly optimizes flow matching in both domains plus teacher-guided motion supervision, with no action-prediction loss, for 20 epochs using AdamW at a learning rate of 10^-4.

Dropout keeps simulation guidance optional

If the student always sees simulation latents, it may simply copy them, including their errors. The paper tests this directly by varying the training-time condition retention probability p over {0, 0.3, 0.7, 1.0} and the inference-time guidance weight w over {0, 0.3, 0.6, 1.0} on the 100-sample Bridge validation set.

Intermediate retention probabilities produced the best observed PSNR and SSIM in this sweep. At p = 0, both metrics stayed unchanged across guidance weights, as expected when no simulation conditioning is used. At p = 1, removing simulation conditioning at inference substantially reduced prediction quality, indicating strong dependence on the condition. By contrast, p = 0.3 and p = 0.7 maintained better performance without conditioning while still benefiting from simulation guidance. The configuration (p, w) = (0.7, 0.6) achieved the highest PSNR and SSIM among the evaluated settings, which is why it is the default choice. For p = 0.7, w = 0.6 outperformed both w = 0 and w = 1, so moderate simulation guidance beat either branch alone.

How much simulation guidance actually helps

A separate sweep varied simulation generation steps Ns, real-video generation steps Nr, and the guidance weight w across all combinations of Ns, Nr in {1, 5, 10, 20} and w in {0, 0.3, 0.6, 1}, giving 64 configurations on the same 100 Bridge validation samples and initial observations. Increasing real-video generation from one to five or ten steps substantially improved prediction fidelity, while going from ten to twenty steps added only small gains. Increasing simulation steps had a much smaller effect and did not consistently improve either metric.

The highest observed PSNR and SSIM occurred at (Ns, Nr, w) = (5, 20, 0.6), reaching 25.090 dB and 0.8585. At w = 0.6, even one simulation step produced results close to larger simulation budgets, suggesting a coarse predicted simulation reference already provides useful guidance. The weight w = 0.6 gave the highest PSNR and SSIM for every tested step combination, and at w = 0 both metrics were invariant to Ns, consistent with the absence of simulation conditioning in the real-video branch. The paper reports that (1, 10, 0.6) reached 25.063 dB PSNR and 0.8575 SSIM in 2.008 s per sample, 0.027 dB below the highest observed value with roughly 47% lower latency than (5, 20, 0.6).

Bridge and InternData-A1 results

On Bridge, SimForcing achieves the best PSNR, SSIM, LPIPS and FVD among the compared methods without external embodied pretraining. Evaluation on InternData-A1 further supports its applicability across robot datasets. These are the paper's headline fidelity and perceptual-similarity results, and they are reported for the compared methods on those benchmarks rather than as a general claim about all robot video prediction.

The two datasets differ in scale and window structure, which matters for reading the numbers. The Bridge subset contains 1,952 robot manipulation trajectories, with windows of 21 consecutive video frames and 20 aligned actions, one action per video transition, and 100 paired validation samples used for evaluation. InternData-A1 uses the Franka manipulation subset with 2,657 training trajectories, windows of 129 steps sampled every four time steps to yield 33 frames, all 128 actions retained, and 80 paired validation samples. In both cases simulated and real videos share the same action sequence, proprioceptive states and language instruction, and both are resized and center cropped to 224x320 pixels.

Downstream policy initialization gains

The paper also tests whether the trained world model is useful beyond video prediction by using it to initialize a vision-language-action model. On LIBERO, this improved the reported average success rate in all three training settings: from 82.6% to 89.2% under action-only training, from 96.5% to 97.5% under unconditional training, and from 98.1% to 98.4% under video plus action training, corresponding to gains of 6.6, 1.0 and 0.3 percentage points.

The largest improvement occurred under action-only adaptation, including gains of 9.6 points on Goal and 15.2 points on Long, which supports the usefulness of the transferred initialization even when the visual expert is frozen. Gains were smaller when both experts received downstream video and action supervision. Video plus action training with this world model reached the highest reported average success rate at 98.4%, compared with 97.6% for the FastWAM reference. The paper notes the improvements are not uniform across suites: in the video plus action setting, gains on Object and Long offset decreases on Spatial and Goal, so the result supports improved downstream transfer without implying improvement on every individual suite.

Where the evidence stops

Several boundaries qualify these results. The inference-step and guidance sweep was a single sweep without repeated-run uncertainty estimates, so small metric differences should not be read as statistically established improvements. The Ns = 20 configurations were evaluated in a separate run, so their timings may be affected by differences in machine load. The recorded w = 0 latency still includes simulation generation and therefore does not represent an optimized configuration that omits unused simulation predictions and the unused conditional branch. In the evaluated implementation, w = 1 uses a single forward pass per real-video sampling step while other weights use the two-branch CFG path.

The downstream evidence is also described by the paper as preliminary. The LIBERO results come from initializing a vision-language-action model with the trained world model, and the paper frames them as suggesting utility for policy learning rather than establishing it. The prediction results are reported on Bridge and InternData-A1 with the specific paired simulation-real subsets and evaluation sample counts described above, so the fidelity and success-rate numbers should be read within those benchmarks and splits.

The priority is guidance strength, not simulation fidelity

The most decision-relevant finding in this paper is not that simulation helps, but how much of it to use. The condition retention probability and the guidance weight interact: p = 0.7 with w = 0.6 gave the best PSNR and SSIM in the conditioning sweep, and w = 0.6 gave the best PSNR and SSIM for every tested step combination. At the same time, p = 1 made the model strongly dependent on simulation conditioning, so removing it at inference substantially reduced quality, while p = 0.3 and p = 0.7 stayed robust without conditioning.

For readers weighing this approach, the practical threshold is therefore the middle of both ranges rather than either extreme. A coarse simulation reference is enough: at w = 0.6, one simulation step came close to larger simulation budgets, and (1, 10, 0.6) traded 0.027 dB of PSNR for roughly 47% lower latency than (5, 20, 0.6). The paper's own framing supports this reading: simulation is used as transferable motion knowledge and as a controllable reference, with classifier-free guidance balancing internalized priors against explicit simulation latents, and only the final student model is needed at inference.

Sources used in this answer