[CVPR 2025] Seoul World Model (SWM): Grounding Generative Simulation in the Real World
Grounding World Simulation Models in a Real-World Metropolis
The paper introduces Seoul World Model (SWM), a city-scale video generative world model grounded in the real-world metropolis of Seoul. By combining retrieval-augmented generation with street-view imagery and a Diffusion Transformer (DiT) backbone, SWM enables high-fidelity, long-horizon video simulation that remains spatially faithful to actual urban locations while supporting diverse camera trajectories and text-prompted scenarios.
TL;DR
Researchers from KAIST and NAVER AI Lab have unveiled the Seoul World Model (SWM), the first city-scale video world model that simulates environments based on actual physical locations. Unlike traditional world models that "hallucinate" random streets, SWM uses retrieval-augmented conditioning on real street-view images to ensure that even after "driving" for hundreds of meters, the generated video remains faithful to the real geometry of the city.
The "Imagined" vs. "Real" World Gap
Current state-of-the-art (SOTA) world models like Sora or GameNGen are impressive at creating visually plausible environments. However, they share a fundamental flaw: they are unanchored. If you tell a model to "drive through a city," it creates a dream-like sequence of generic buildings.
The authors of SWM ask: What if a world model could render a city that actually exists? This is remarkably difficult because:
- Data Sparsity: Street-view images (like Google Maps) are captured at discrete intervals (5-20m), not as continuous video.
- Temporal Ghosting: If you retrieve a reference image of a street, it might have a bus parked there that isn't in your current simulation. The model must learn to ignore these "transient" objects.
- Drift: Autoregressive generation (predicting frame-by-frame) inevitably leads to "drift" where the world slowly melts or changes into something unrecognizable.
Methodology: The Architecture of Grounding
SWM addresses these challenges through a sophisticated retrieval-augmented Diffusion Transformer (DiT) framework.
1. Cross-Temporal Pairing: Learning Persistence
To solve the "temporal ghosting" problem, the authors use cross-temporal pairing. During training, the reference images and target videos are sampled from the same location but at different times. This forces the model to ignore the cars and pedestrians (which change) and focus strictly on the buildings and road layouts (which persist).
2. The Model Architecture
SWM employs two complementary pathways for grounding:
- Geometric Referencing: Warping reference images into the target viewpoint to provide a spatial "skeleton."
- Semantic Referencing: Injecting original reference images into the transformer’s latent sequence to provide high-fidelity "textures."
Figure: The SWM architecture showing the dual referencing pathways and the retrieval mechanism.
3. Virtual Lookahead Sink: The "North Star" for Video
Standard video models use the first frame as an "attention sink" to maintain consistency. But if you’ve travelled 500 meters, that first frame is irrelevant. SWM introduces the Virtual Lookahead Sink. It retrieves a street-view image of where the camera will be in the future and injects it as a fixed anchor. This acts as a "virtual destination," preventing the model from drifting off-course.
Experiments: Real-World Performance
SWM was evaluated not just in Seoul, but also in "zero-shot" settings in Busan and Ann Arbor, showing remarkable generalization.
Figure: SWM following a long trajectory in Seoul. Notice how the generated environment (bottom) remains aligned with the real map layout (top).
Key Breakthroughs:
- Stability: While baselines like DeepVerse or Aether often collapse into blur after 100 meters, SWM maintains sharp structures.
- Controllability: Users can prompt the model with "flooded streets" or "golden sunset," and the model will render those effects on top of the real urban geometry of Seoul.
- Diversity: Thanks to synthetic data from the CARLA simulator, SWM isn't limited to car-mounted views; it can simulate pedestrian paths and "free camera" movements.
Critical Analysis & Conclusion
The genius of SWM lies in its Retrieval-Augmented Architecture. By realizing that we don't need to store the whole world in the model's weights—but can instead "look up" the world from map databases—SWM achieves a level of scale and precision previously thought impossible for generative models.
Limitations: The model is still sensitive to the quality of street-view metadata. If the GPS data is noisy, you might see "glitches" where cars suddenly blink out of existence.
The Future: SWM represents a shift from "Creative AI" (making things up) to "Simulation AI" (reproducing reality). This is a critical building block for the next generation of Autonomous Driving simulators and the industrial Metaverses.
Takeaway
SWM proves that your "World Model" doesn't have to be a hallucination; with the right retrieval and anchoring mechanisms, it can be a digital twin of the real world.