What does scaling 30,000 hours of ego-centric video fail to teach world models?

Scaling helps the agent and the world unequally

The study asks how far scaling ego-centric human video takes a learned world model — a generative model that predicts future video frames conditioned on an agent's actions, offered as an alternative to physics-based simulators. The authors train Cosmos 3 video world models on nested subsets of a 30,000-hour first-person manipulation corpus (1.15M clips, 32,051 tasks, 1,106 scene categories, 14,019 participants) and evaluate on a held-out out-of-distribution benchmark with different capture hardware, sites and participants.

Increasing training data by 100x improves both things the authors measure, but not equally. Agent fidelity — how well the model renders the hands and body — climbs steadily toward usable levels. Object fidelity — how well it renders the manipulated objects — stays far lower and improves more slowly. The gap between modeling the agent and modeling the agent's effect on the world actually grows with data scaling, which is the paper's central problem.

The metrics are structural, not perceptual. SCS (structural consistency score) segments and tracks key elements through predicted and ground-truth video and averages mask IoU, reported separately for hands and for manipulated objects. LPIPS serves as a perceptual control so that structural differences can be checked against image-quality differences. Because mask tracking is imperfect, the authors estimate an achievable ceiling of 0.93 for the agent and 0.90 for objects on ground-truth video — both well above what the models reach, so the reported differences reflect the models rather than the metric's limits.

Agent fidelity saturates well before 30,000 hours

Agent fidelity gains 0.12 SCS between 300 and 3k hours but only 0.06 between 3k and 30k. Fitting a saturating curve of the form S∞ − cD^−α, standard in scaling-law analyses, places the agent asymptote at 0.811. That is the first sign that data volume alone is not the binding constraint on this axis.

Skeleton conditioning makes the point sharper. This design supplies the agent's projected body configuration in the camera frame, so the model no longer has to infer body pose from action tokens. With only 300 hours, the skeleton-conditioned Nano variant reaches the agent fidelity the baseline attains at roughly 15k hours — about a 50x reduction in data — and then gains just 0.06 SCS over the remaining two orders of magnitude.

Crucially, conditioning changes when the agent's limit is reached, not what it is: the two designs converge to within about 0.01 SCS of each other (0.811 and 0.800). The authors test whether conditioning is a shortcut that spares the model visual inference useful for interaction modeling by randomly dropping the skeleton during training; they find no evidence of one. Saturating the agent axis early therefore gives them an instrument: with the agent held near its ceiling across the data ladder, the object curve can be read without being confounded by improvements in agent fidelity.

Object fidelity converges to a much lower ceiling

Measured with the agent held near its ceiling, object fidelity improves by only 0.09 SCS across two orders of magnitude, and unlike the baseline it decelerates: 0.056 between 300 and 3k hours, then 0.035 between 3k and 30k. The curve is now in its saturating regime and its asymptote is identifiable at 0.565, of which the model has already realized 93% at 30k hours.

Normalized by the maximum the metric can register for each category, the agent converges to 86% of its measurable range and objects to 63%. The authors state the implication directly: the limitation is not that object dynamics improve slowly with data, but that the level they improve toward lies far below the agent's.

Model size is not a quick fix either. Scaling the model by 4x (the Super variant) has virtually no effect on agent fidelity but raises the object ceiling by 0.034 SCS. At that rate, closing the remaining 0.205 to the agent ceiling would require an implausible amount of further model scaling — the authors compute the requirement and conclude it does not work quantitatively, even though it supports the scaling-centric view qualitatively.

Retargeting supervision raises the ceiling, not the slope

The authors then introduce a supervision scheme that shifts model capacity from scene appearance toward object dynamics. It combines dynamic noise scheduling with loss reweighting, applied at both training and inference time. In a regional supervision ablation at 10k hours, this combination reaches the highest object fidelity (0.532 ± 0.003) without sacrificing perceptual quality; prior auxiliary objectives such as PhysisForcing (0.515) and reweighting alone (0.516) sit close to the 0.513 skeleton baseline.

Across the full data ladder the method exceeds the skeleton baseline at every budget, reaching 0.546 against 0.527 at 30k hours — a 3.7% improvement — and fitting the same saturating form places its asymptote above the baseline's. The gain is roughly constant across budgets: the ceiling is raised, not the rate at which it is approached, and it remains far below the agent's. The authors conclude that reaching a usable level of interaction fidelity will require more substantial changes to how world models are trained.

Contact and force are not visible in pixels

The paper offers two explanations for the low object ceiling. The first is irreducible uncertainty: how an object responds to contact depends on mass, friction and stiffness, none of which is perfectly recoverable from a single frame. The second is that the quantities that matter most are not observable in pixels at all. An object's geometry is only partially visible, and the parts most relevant to an interaction — those under the hand, at the point of contact — are exactly the parts the hand occludes. Contacts and forces are never visible.

These are precisely what a physics simulator represents exactly and can supply as supervision, while remaining poor at the complex dynamics that motivated learned world models in the first place. The authors frame the two paradigms as strongest where the other is weakest, and point to work distilling simulators into differentiable generative 3D particle models as a possible way to combine them. This is the authors' interpretation of the mechanism, not a separately measured result in the paper.

The pattern transfers to humanoid modeling

To test whether the conclusions travel, the authors fine-tune Cosmos 3 Nano on ground-truth demonstrations from 11 tasks in a simulated bimanual humanoid benchmark, conditioning on a single start frame and a 138-dimensional humanoid state trajectory and predicting the following 16 frames. They compare released weights that never saw the ego-centric data, a matched checkpoint pre-trained on 30,000 hours without skeleton conditioning or the new supervision scheme, and a checkpoint pre-trained with both. All evaluations run on policy rollouts rather than demonstrations.

Ego-centric human pre-training transfers, improving every measure and most clearly hand SCS (+0.07). The conditioning and supervision design contributes more than twice as much on top of it. The pattern matches the human-video findings in one respect and departs in another: gains are larger for the agent than for the objects it manipulates (+0.17 versus +0.06), but object fidelity is high in absolute terms for every variant (0.70–0.79). The authors attribute that to the simulated environment, whose objects are rigid and relatively large, so the conditions that make object dynamics hard in ego-centric human video are largely absent.

Success agreement — how often the model's rendered rollouts support the same success verdict as the simulator within the critical 17-frame window — follows the same ordering with smaller margins and is less diagnostic, because a single number records that agreement differed without indicating where. SCS localizes the residual error to the objects: over the prediction horizon, hand SCS for the full method falls from 0.89 at frame 2 to 0.85 at frame 16, while object SCS falls more than twice as far, from 0.82 to 0.71.

Where the ceilings are optimistic and where they hold

The data ladder spans two orders of magnitude and couples data volume with optimization budget, so it establishes where fidelity is heading under joint scaling, not the effect of either factor alone. Because smaller budgets receive proportionally fewer optimization steps, they are further from convergence; training each to convergence would raise the left of the curve more than the right and therefore lower the fitted asymptote. The reported ceilings are, in the authors' words, optimistic. They also hold only insofar as the fitted regime continues beyond 30k hours.

Both training and evaluation sets come from a single collection protocol, and the evaluation set is limited in scale, which constrains how far the conclusions generalize to ego-centric video as a whole. The experiments also study generative world models specifically; whether latent-predictive models capture interaction dynamics more readily remains open and cannot be settled with these fidelity metrics, since those models do not render pixels.

The practical priority that follows is specific to this paper's comparison. If the goal is a world model whose predictions can stand in for a simulator during policy training or planning, the binding constraint is object-interaction fidelity, not agent rendering — and the evidence says that constraint is set by training design rather than by hours of ego-centric video. The humanoid transfer results add a second threshold: conclusions about interaction fidelity depend heavily on how hard the evaluation setting is, since rigid, large simulated objects produced object scores of 0.70–0.79 where ego-centric human video produced a 0.565 ceiling. A benchmark with easy objects will make a world model look ready when the ego-centric setting says it is not.

Sources used in this answer