[CVPR 2026] LMGenDrive: Bridging Perception and Imagination for Robust End-to-End Driving

TL;DR

LMGenDrive is the first autonomous driving framework that truly merges the "Brain" (LLM reasoning) with the "Eyes/Imagination" (Generative World Models). By predicting both actions and future video frames simultaneously, it treats driving not just as a mapping task, but as a continuous process of understanding and anticipation. It achieves a 62.2 Driving Score on the CARLA LangAuto benchmark, significantly surpassing previous leaders.

Problem & Motivation: The Gap Between Reasoning and Imagination

Building an Autonomous Vehicle (AV) that can handle "long-tail" scenarios—those rare, chaotic moments like a pedestrian suddenly darting out in a storm—is the industry's ultimate hurdle.

Currently, research has split into two camps:

1. MLLM-based Driving: Uses models like Llama to "think" about the scene. They are great at following instructions but often "blind" to how the scene will physically evolve over the next 5 seconds.
2. Generative World Models: These models can "imagine" high-fidelity videos of the future but lack the semantic "knowledge" to follow complex human instructions or reason about abstract traffic rules.

LMGenDrive's Insight: Human drivers don't just see; they anticipate. We use our internal world model to simulate "What if?" before we steer. LMGenDrive implements this biological intuition in a unified transformer-diffusion architecture.

Methodology: A Unified Cognitive Architecture

The architecture of LMGenDrive is a sophisticated pipeline that transitions from raw pixels to semantic queries, and finally to both actions and videos.

1. The Multi-modal Brain (LLM)

At its core, a Vicuna-7B model processes navigation instructions and visual tokens. Instead of feeding every pixel, the system uses a Q-Former to compress visual features into a handful of "context tokens." Two specific sets of learnable queries are introduced:

• Action Queries: Used to predict waypoints and control signals.
• World Queries: These capture the "vibe" and physics of the evolving scene to guide the video generator.

2. The Multi-View World Generator

The world model isn't just a generic video maker. It is a Diffusion-based U-Net that is conditioned by the LLM’s world queries and the previous frame's multi-view images. This ensures that the "imagined" future is both physically plausible and semantically aligned with the driver's intent.

3. Three-Stage Training Strategy

To ensure stability (as training LLMs and Diffusion models together is notoriously difficult), the authors use a curriculum approach:

• Stage 1: Pretrain the vision encoder (BEV perception tasks).
• Stage 2: Jointly train LLM and Video Gen for single-step prediction.
• Stage 3: Long-horizon training (2-3 steps) to ensure temporal consistency and reduce "drift" in imagination.

Experiments: Superior Performance and Realism

LMGenDrive was tested on the LangAuto benchmark in the CARLA simulator. This benchmark is particularly difficult because it requires the agent to follow natural language commands (e.g., "Turn right at the next light") across diverse urban environments.

SOTA Comparison

As shown in the table below, LMGenDrive creates a massive gap between itself and previous models like LMDrive and BEVDriver.

Visualizing the Imagination

The model maintains impressive spatial consistency. In the qualitative examples, the "imagined" left, front, and right views align perfectly as the vehicle navigates an intersection.

Deep Insight: Why Does Imagination Help Planning?

The most striking finding in the ablation study (Table 2) is that removing the world generator tanks the Driving Score from 62.2 to 53.4.

Why? Because the act of predicting pixels forces the LLM to learn a much deeper representation of space and time. To generate a realistic video of a car turning, the LLM must truly understand the vehicle's dynamics and its relationship to other agents. The "World Queries" act as a regularizer that prevents the LLM from making "hallucinated" decisions that don't respect physical constraints.

Conclusion & Future Work

LMGenDrive represents a major step toward Embodied AGI. It proves that by unifying understanding (LLM) and imagination (World Model), we can create driving agents that are not only smarter but more robust.

Limitations: While performant, the model faces "temporal drift" during very long-horizon autoregressive generation (128+ frames), where the imagined world slowly dissolves. Solving this "entropy" of imagination will be the key to 100% reliable autonomous driving in the future.