Myriad: Scaling Future Envisioning via Sparse, Step-wise Trajectory Diffusion
Envisioning the Future, One Step at a Time
This paper introduces Myriad, an autoregressive diffusion model that predicts open-set future scene dynamics as step-wise inference over sparse point trajectories rather than dense pixels. It achieves state-of-the-art predictive accuracy on the new OWM benchmark while enabling sampling speeds orders of magnitude faster than dense video simulators.
TL;DR
Myriad is a 665M-parameter autoregressive diffusion model that abandons the "visual tax" of generating pixels to focus exclusively on sparse point trajectories. By simulating futures step-by-step, it can unroll thousands of plausible, physically consistent trajectories in the time a traditional video model generates just one, enabling complex zero-shot planning and superior open-set motion prediction.
Background Positioning: This work represents a shift from appearance-centric generative models (Sora, SVD) to dynamics-centric reasoning, occupying a unique niche between rigid physics engines and heavy video simulators.
The Problem: The "Visual Tax" and the Chaos of One-Step Prediction
Predicting the future is a massive search problem. If a billiard ball is struck, there are infinite potential outcomes based on microscopic variations in friction or contact. Current state-of-the-art (SOTA) video models like Wan2.2 or CogVideo-X attempt to solve this by "painting" every pixel. This incurs a massive visual tax: the GPU spends more time generating the texture of the billiard cloth than calculating the trajectory of the ball.
Furthermore, many trajectory models attempt "one-shot" prediction—jumping from frame 0 to frame 100. This is physically impossible for complex scenes where the outcome is a result of a sequential chain of interactions.
Methodology: The Dynamics-Centric Architecture
The core insight of Myriad is to model the distribution of possible future motion over sparse points.
1. Motion Tokens & Randomized IDs
Instead of pixel patches, the model consumes "Motion Tokens." These tokens fuse spatial features (retrieved via bilinear sampling from a DINOv3 encoder) with a unique Randomized Trajectory Identifier. By drawing IDs from a unit sphere , the model remains nearly orthogonal across points, allowing it to scale to an arbitrary number of query points without retraining.
2. The Scale Cascade & Flow Matching
Motion in the real world is "heavy-tailed"—objects mostly move a little, but occasionally jump a lot. Myriad uses a Posterior Flow Matching (FM) Head with a "Scale Cascade." This tanh-saturated input stack allows the network to handle tiny sub-pixel tremors and massive leaps simultaneously without numerical instability.
3. Fast Reasoning Blocks
To maximize throughput, the authors implemented Fast Reasoning Blocks. These fused layers combine self-attention (between trajectories) and cross-attention (to the image) into a single residual pass, cutting kernel launches in half and doubling sampling speed.
Experiments: Planning and Benchmarking
The authors introduced OWM (Open-World Motion), a benchmark of 95 in-the-wild videos with static cameras to measure physical plausibility.
Results vs. Video Models
In a head-to-head comparison with models like Stable Video Diffusion (SVD), Myriad was not only more accurate in a "best-of-5" attempt but dominated in "Best-within-5min" tests. Because Myriad can sample 2200 futures per minute (vs. SVD's 0.7), it can exhaustively explore the probability space of an interaction.
Zero-Shot Planning (Billiard Shot)
The most striking result is in counterfactual planning. To sink a billiard ball, a model must test hundreds of "what if" scenarios. Myriad achieved 78% accuracy, nearly matching an Oracle simulator (84%), while video-based world models collapsed to <20% because they simply couldn't generate enough "hypotheses" in time.
Critical Analysis & Takeaways
Why it works: Myriad succeeds because it respects the incremental nature of uncertainty. By factorizing the future into short, locally predictable steps (), it avoids the error accumulation that plagues one-shot models and the compute-sink of video models.
Limitations:
- Static Camera: The current formulation assumes a stationary observer. While 3D re-projection can mitigate this, it is a significant constraint for mobile robotics.
- Tracking Bias: It relies on "pseudo-ground truth" from trackers like TAPNext, meaning it can only be as good as the labels it harvests.
Conclusion: Myriad is a compelling argument for Sparsity in Reasoning. As we move toward autonomous agents, the ability to rapidly simulate 1,000 possibilities in "motion space" will likely be more valuable than generating a single high-fidelity video of a failure.