World in World: Training-Free Visual Evidence Orchestration for Frozen Video World Models

Core Snapshot

TL;DR

World in World, abbreviated WiW, is a training-free inference-time interface for autoregressive video world models. It converts four heterogeneous evidence channels, namely source-video observations, target-view scene projections, rendered subject geometry, and retrieved generated history, into clean visual states annotated with camera pose, event time, spatial support, and denoising-stage activation. A frozen causal video backbone then reads these states through its native self-attention, while correspondence-guided attention routing localizes geometrically meaningful source tokens and evidence-wise attention CFG independently regulates each channel's attention response. On camera-controlled rerendering over DAVIS and OpenVid-1M, Table 1 reports VBench Overall 85.192, TransError 0.068622, RotError 2.8326 degrees, PSNR 23.1511, and SSIM 0.787205, placing WiW ahead of the compared baselines on the reported aggregate score and both camera error measures.

Background Positioning

This paper is best understood as an interface and orchestration contribution rather than a new video generator. It builds on the publicly released causal-fast checkpoint of LingBot-World 2.0 and freezes all pretrained parameters, as stated in Section 4.1. Its novelty sits between camera conditioning modules, source-video rerendering systems, and long-term memory mechanisms: instead of learning a new control pathway for each requirement, WiW claims that existing self-attention over clean visual states is already a generic control port. This is a strong architectural claim, but its practical force depends on the surrounding inference-time machinery: depth estimation, projection, avatar reconstruction, point tracking, retrieval, and attention guidance.

Problem and Motivation

Autoregressive video world models already generate future observations from past visual states, but controllable exploration asks for more than plausible motion. A user may move the target camera along a trajectory while expecting the underlying recorded event to progress in time. The generation must simultaneously preserve the source event, place content correctly under the new camera, complete regions not visible in the source, and recover appearance after the camera revisits earlier areas. These four demands are not redundant. A source frame provides content but not target location. A projected frame provides target location but may be wrong under occlusion or depth error. Geometry provides completion cues but only for renderable subjects. History retrieval recovers evicted states but requires a selection rule that avoids redundant or misleading references.

Existing methods usually attack subsets of this problem with task-specific modules or extra training. The paper reviews camera conditioning as rays, poses, projected features, or geometric proxies, and notes that source-video rerendering and long-horizon memory often require dedicated pathways or jointly trained control fields. The failure mechanism is therefore representational and temporal. A new evidence type can be injected, but the model may not know where it should act, when it should act, and how strongly it should act. Naively concatenating tokens makes attention ambiguous under large viewpoint change, repeated texture, dynamic motion, and occlusion. Naively scaling all evidence equally risks overguiding from imperfect geometry and suppressing the pretrained model's generative prior.

The authors' central intuition is simpler: a causal video model with a clean-state cache already reads visual evidence through self-attention. Initial observations and finalized outputs are already represented as clean visual states with camera and temporal information. If external controls are converted into the same format, then no new learned adapter is strictly required. This reframes flexible control as visual evidence construction and attention orchestration. The paper does not prove this as a theorem, and it does not claim that every control can be expressed through clean states. Instead, it provides an empirical and architectural argument: pretrained video models process visual tokens, camera/time encodings, and attention over cache slots, so the clean-state pathway is a natural shared entry point.

Core Mechanics: The Architecture of Training-Free Control

The clean-state cache as a shared control port

WiW first abstracts every evidence source into a common tuple. At denoising step $s$, raw evidence $E_b$ from channel $b$ and target camera trajectory $C$ are mapped by a converter ${\Psi }_b$ to visual content, camera and time metadata, spatial support, and channel activation.

Here, $X_b$ is the visual content of the evidence, $P_b$ records per-frame camera intrinsics, poses, and event-time indices, ${\omega }_{bj}\in [0,1]$ is the token-level spatial support weight for evidence token $j$, and ${\gamma }_b^{(s)}\in \{0,1\}$ indicates whether channel $b$ is active at denoising step $s$. The tuple separates four questions that are frequently collapsed in conditioning literature: what the evidence contains, where it belongs in the camera and event timeline, which regions can trust it, and which denoising stages should see it. Compared with a cross-attention adapter, this representation is intentionally native to the backbone: the paper feeds clean visual states into the frozen model with diffusion timestep $t=0$, extracts per-layer keys and values, and caches them as clean K/V features. During target-chunk generation, current queries read these cached features through self-attention. If ${\gamma }_b^{(s)}$ were omitted, every evidence channel would either participate at all stages or require an ad hoc schedule. If ${\omega }_b$ were absent, unreliable occluded regions would be given the same attention mass as reliable visible regions. The tuple is therefore not decorative; it is the mechanism that lets multiple evidence streams share the frozen attention stack.

The native attention state is also structured. Section 3.1 describes 18 latent-frame-state slots: six source anchors, eight recent-history states, and four current-chunk states. Source-video evidence occupies the source-anchor slots, while other evidence is attached as temporary auxiliary attention blocks that are removed once the current chunk is finalized. All blocks reuse the frozen backbone's query, key, and value projections and preserve native text and camera conditioning pathways. The paper also sets temporal coordinates for rotary position embeddings and applies the corresponding rotations to queries and keys, making relative time explicit inside attention. This design is important because it avoids permanently polluting the autoregressive state with transient control tokens. Temporary blocks can influence the current chunk without becoming part of the rolling history unless they are finalized outputs.

Complementary evidence channels answer different gaps

The second design layer asks which evidence should be constructed. WiW does not treat all channels as alternatives; it treats them as complements. Source-video observations provide appearance references, but they do not place content under the target camera. Target-view projections solve placement. Rendered geometry solves newly exposed subject surfaces. Retrieved history solves evicted long-range state.

Target-view scene evidence starts from source RGB and estimated depth. The paper uses DepthCrafter for depth estimation and back-projects source pixels into a shared 3D coordinate system before reprojecting them from the source camera to the target camera.

In this expression, $I_{r,m}^{\mathrm{src}}$ is source RGB temporally aligned with target frame $m$, $D_{r,m}^{\mathrm{src}}$ is the estimated source depth, $C_{r,m}^{\mathrm{src}}$ is the source camera, and $C_m$ is the target camera with intrinsics $A_m$ and pose $T_m$. The outputs are projected target-view RGB $X_{r,m}^{\mathrm{proj}}$, a binary visibility mask $M_{r,m}^{\mathrm{proj}}$, and target-camera depth $Z_{r,m}^{\mathrm{proj}}$. The projected evidence is then encoded through the same clean-state interface, with visibility and geometric reliability converted into token support weights. This matters because projection is useful only where geometry is trustworthy; otherwise it can force the model to follow warped artifacts. If the support mask were ignored, occlusion holes and depth discontinuities would become confident wrong layout references.

Rendered geometry evidence addresses a different failure mode. When large camera motion exposes subject surfaces never observed in the source video, projection cannot help because those surfaces are absent. WiW instead builds a renderable subject representation and asks it for shape and appearance proposals at the current event time and target camera.

Here, $G_m$ is the subject state at event time $m$, and $C_m$ is the target camera. The outputs are rendered RGB, a geometric support mask, and target-camera depth. For human subjects, the paper specifies $G_m=(A,{\Theta }_m)$, where $A$ is an avatar reconstructed with LHM++ from sharp, minimally occluded full-body source crops, and ${\Theta }_m$ contains per-frame SMPL-X parameters driving that avatar. The geometry is aligned to the projected scene's coordinate frame and depth scale using source-view correspondences. It also compares rendered subject depth with projected scene depth to suppress unreliable background projections near subject boundaries. This channel is therefore not a generic "add 3D" trick. It is a targeted completion cue for subjects whose unseen surfaces need pose-consistent structure, while the pretrained video model supplies the final texture and appearance prior.

Generated history retrieval is the third complementary channel. The rolling cache retains only recent states, so an earlier region visited by the camera may no longer be available after eviction. WiW maintains a rollout-wide history bank. For each evicted finalized state, it archives per-layer clean K/V together with camera pose and temporal index. For a target chunk, it ranks historical states by target-view surface coverage and viewing-direction compatibility, then selects a bounded, diverse top-K subset. The paper does not give a formal retrieval objective in equation form; the selection rule is described verbally. The model then reads the retrieved K/V through temporary auxiliary attention blocks. This turns retrieval into pseudo-history: the backbone does not see a memory module, but sees clean visual states associated with past camera and time coordinates.

Routing and regulation decide when evidence helps

Having accessible evidence is not enough. The current denoising queries must read the right tokens. Appearance-based attention can fail under viewpoint change, dynamic motion, and repeated texture. WiW introduces correspondence-guided attention routing to bias attention toward geometrically matched source-video tokens.

The first term is the usual scaled dot-product attention logit for query $i$ and key $j$, where ${\mathbf{q}}_i$ and ${\mathbf{k}}_j$ are attention features and $d_h$ is the head dimension. The added term $\log {\beta }_{ij}$ is a geometric prior for a valid correspondence between query $i$ and source-video key $j$, with ${\beta }_{ij}>0$. The use of the logarithm is meaningful: adding log weight to an attention logit is equivalent to multiplying the unnormalized attention weight by ${\beta }_{ij}$ before normalization. If ${\beta }_{ij}$ is large, the matched key is promoted; if it is small, the correspondence suppresses that path. If no valid geometric correspondence exists, the query is not routed through this channel. This mechanism is local and sparse by design. It does not ask the model to relearn cross-view matching; it supplies a point-tracking plus depth plus camera geometry prior into the attention score.

The second regulatory layer is evidence-wise attention CFG. Different channels have different reliability across space and denoising stages. Using all evidence with the same strength makes it hard to balance adherence to projection, geometry, history, and the native generative prior. Inspired by classifier-free guidance and normalized attention guidance, EWA treats each evidence channel's attention response separately.

Let ${\mathbf{o}}_0$ denote the native-block-only attention response, and let ${\mathbf{o}}_e$ denote the jointly normalized response when native tokens and evidence $e$ are both attended. The guidance strength $g_e\ge 0$ controls how much of the complementary direction is added. The term ${\mathbf{o}}_e-{\mathrm{Proj}}_{{\mathbf{o}}_0}({\mathbf{o}}_e)$ removes the component of the evidence-conditioned response that lies along the native response. EWA then amplifies only the residual, and only when the cosine similarity is positive. This is a subtle but important design choice. If the method merely amplified ${\mathbf{o}}_e$, it would also reinforce directions already present in the native model, potentially over-emphasizing the model's own output rather than the evidence's new information. If the cosine gate were removed, weakly aligned evidence could still be injected. If $g_e$ were set to zero, EWA reduces to the joint response. If all $g_e$ were identical, the method would lose the channel-wise regulation that motivates the acronym. The paper further bounds output magnitude using the native response norm as a reference, preventing excessive guidance from destabilizing generation. EWA operates within the same denoising forward pass and adds no network function evaluations for guidance itself.

Experiments and Evidence

Main rerendering comparison

Section 4.1 states that WiW is instantiated on the public causal-fast checkpoint of LingBot-World 2.0 with all pretrained parameters frozen. The evaluation focuses on camera-controlled video rerendering from DAVIS and OpenVid-1M under diverse viewpoint changes. Baselines include ReCamMaster, TrajectoryCrafter, WorldForge, InSpatio-World, UniWorld-View, and CameraAnything, each using its official implementation and recommended configuration. The paper notes that ReCamMaster and TrajectoryCrafter generate 81 and 49 frames respectively due to official implementations, while the remaining methods use their recommended frame counts; generated videos are then aligned in camera angles and frame count. Camera errors are estimated independently with Depth Anything 3 and ViPE, and reported TransError and RotError are averaged over the two estimators.

Table 1 reports the main quantitative comparison. The extracted table gives VBench Overall, camera errors, and the available fidelity entries PSNR and SSIM; the LPIPS column is not cleanly recoverable from the supplied text, so it is not quoted here.

The table supports three main claims. First, WiW has the highest VBench Overall score, 85.192, compared with the second best UniWorld-View at 84.295. Second, it has the lowest reported camera errors: TransError 0.068622 and RotError 2.8326 degrees. The TransError advantage over UniWorld-View is numerically tiny, from 0.068705 to 0.068622, while the RotError advantage is more visible, from 4.1958 to 2.8326 degrees. Third, it improves the reported fidelity columns, reaching PSNR 23.1511 and SSIM 0.787205, above UniWorld-View's 22.4735 and 0.770554. The paper also lists the seven VBench sub-dimensions. From Table 1, WiW obtains the best reported values in Aesthetic Quality 56.556, Imaging Quality 68.004, Subject Consistency 88.948 by a narrow margin over UniWorld-View's 88.946, Background Consistency 92.627, and Motion Smoothness 97.749, while Dynamic Degree is tied at 98.750 with TrajectoryCrafter. Temporal Flickering is not best; ReCamMaster reports 95.178 compared with WiW's 93.711.

The qualitative comparison in Figure 3 is important because camera error and VBench can hide different failure types. A method may follow the trajectory but drift in appearance, or preserve texture but miss target-view layout. WiW's claimed advantage is that projection provides placement, geometry guides newly exposed subject regions, and retrieved history helps with revisits. The figure illustrates these cases under diverse camera motion. It does not quantify failure frequency, so the robustness argument remains partly visual.

Ablation: projection is dominant, routing and guidance are refinements

Table 2 ablates individual components on DAVIS. It reports five VBench dimensions plus camera errors.

The ablation reveals the hierarchy of evidence. Target-view warp is the dominant channel. Removing it raises RotError from 1.7823 to 6.1158 degrees and TransError from 0.053291 to 0.581847. The paper describes these as approximately 3.4 times and 10.9 times degradation, matching the table values. This is strong evidence for the spatial layout argument in Section 3.2: without target-view projection, the model lacks the explicit placement needed for accurate camera following.

By contrast, CGAR and EWA are refinements. Removing CGAR changes RotError only from 1.7823 to 1.7828 degrees, while removing EWA changes it to 1.9944 degrees. Removing both increases RotError to 2.0346 degrees and TransError to 0.056937. The VBench differences among the full method and the routing or guidance ablations are also small. This suggests that correspondence routing and evidence-wise guidance improve consistency and suppress overguidance, but do not replace the geometric layout provided by target-view warp. Removing source Plucker conditioning has a modest camera penalty, increasing RotError to 1.8811 and TransError to 0.056887, supporting the claim that both target-view layout and source-view camera information matter.

Figure 6 complements the numerical ablation with cases that Table 2 does not fully capture. It shows routing and guidance helping preserve subject appearance and background structure, rendered body geometry guiding newly exposed subject regions, and historical retrieval helping preserve appearance and layout after earlier states have left the rolling cache. These are qualitative demonstrations of the paper's central evidence taxonomy. They make the mechanism intuitive, but they do not provide a quantitative revisit benchmark or a systematic failure-rate comparison.

Evidence quality assessment

The strongest evidence in the paper is the main Table 1 result: WiW is competitive across multiple metrics under a unified source-video and target-camera protocol. The ablation Table 2 further identifies target-view projection as the dominant mechanism and validates that routing and guidance reduce camera error. The design choices correspond to observable effects: projection helps placement, geometry helps subject completion, retrieval helps revisits, and EWA and CGAR help control and localization.

However, the evidence quality has boundaries. First, the quantitative evaluation is limited to two datasets, one backbone checkpoint, and camera-controlled rerendering as the main task. The paper claims broader applications such as bullet-time generation, video stabilization, video editing, human-motion transfer, and K/V sharing between two instances of the same frozen model, but these are presented qualitatively in Section 4.4 rather than as benchmark tables. Second, camera errors rely on estimated trajectories from Depth Anything 3 and ViPE rather than direct ground-truth pose comparison, so estimator bias may affect the ranking. Third, the ablation is on DAVIS and only five VBench dimensions, and the differences between full method and routing or guidance variants are small. Fourth, the paper does not report inference overhead from DepthCrafter, tracking, avatar reconstruction, SMPL-X optimization or inference, retrieval ranking, or temporary auxiliary attention blocks. The "training-free" claim is convincing, but "inference-free cost" is not established.

Deep Insights and Summary

The paper's real contribution is not that each evidence component is new in isolation. Camera conditioning, depth projection, geometric guidance, memory retrieval, attention priors, and classifier-free-style modulation all have prior art. The contribution is that WiW organizes them as visual evidence flowing through the native attention interface of a frozen causal video model. That organization yields a practical principle: for pretrained world models, control can be expressed as camera and time labelled clean states plus attention regulation, rather than as a separate trainable adapter for every new control mode. The method preserves the backbone's pretrained visual prior while allowing external evidence to enter selectively.

This reframing also clarifies why the method works. Projection and geometry provide complementary spatial constraints: where known content should appear, and what should appear when the source view has no information. Retrieval provides temporal continuity beyond the rolling cache. CGAR reduces local cross-view ambiguity by using persistent point identities and geometry. EWA prevents different channels from competing destructively by regulating their attention-response residuals. The system is thus more than a token injection technique; it is a full evidence lifecycle: construct, annotate, localize, regulate, and discard temporary evidence without corrupting the finalized state.

The limitations are equally specific. WiW depends on external evidence pipelines whose failures can propagate: DepthCrafter depth errors affect projection masks, LHM++ and SMPL-X reconstruction or alignment errors affect geometry evidence, point tracking errors affect routing, and the unformalized retrieval ranking may miss useful evicted states. The method is currently demonstrated on a single frozen backbone, LingBot-World 2.0 causal-fast, so its transfer to other causal video models, longer horizons, or non-camera interactive controls remains unquantified. The paper also does not provide a formal optimization objective for retrieval or a full report of guidance hyperparameter search. Finally, because EWA adds no additional NFE but still uses temporary auxiliary blocks and external geometry, latency, memory, and robustness trade-offs are not fully characterized.

A productive next step would be to evaluate revisit consistency quantitatively rather than qualitatively. A benchmark could measure whether evicted regions are recovered with lower LPIPS or higher SSIM after retrieval, while disabling the history bank and keeping projection and geometry fixed. Another useful extension is to replace hand-designed evidence channels with a learned evidence selector that predicts channel reliability per token from geometric and temporal cues. For practical engineering, WiW suggests that frozen video world models can become more controllable through cache and attention interfaces, but it also warns that the new complexity is moved into robust visual-evidence orchestration.