[arXiv 2026] LAD-Drive: Solving the Intention-Action Gap in LLM-Based Autonomous Driving
LAD-Drive: Bridging Language and Trajectory with Action-Aware Diffusion Transformers
LAD-Drive is an end-to-end autonomous driving framework that bridges the gap between discrete Multimodal Large Language Model (MLLM) reasoning and continuous trajectory planning. It uses a decoupled architecture featuring an Action Decoder for meta-action distribution and a Diffusion Decoder for trajectory refinement, achieving a new SOTA on the LangAuto benchmark with a 68.2 Driving Score.
TL;DR
LAD-Drive introduces a generative framework that bridges high-level linguistic reasoning and low-level physical execution. By disentangling semantic intent (probabilistic meta-actions) from spatial planning (diffusion-based refinement), it achieves a massive +59% improvement in Driving Score on the LangAuto benchmark, effectively eliminating the "mode averaging" problem found in traditional regression-based planners.
Background: The Modality Mismatch
The integration of Multimodal Large Language Models (MLLMs) into driving has promised "human-like" reasoning. However, a fundamental bottleneck remains: How do we translate a discrete word like "Turn Left" into a continuous, safe, and kinematically feasible trajectory?
Prior works generally fall into two (flawed) camps:
- Direct Autoregression: Predicting waypoints as text tokens. This is physically nearsighted and often produces "hallucinated" coordinates.
- Coupled Regression Heads: Using an MLP to map LLM features to waypoints. This suffers from "mode averaging"—if a model sees both "turn left" and "go straight" as valid, it might predict a path straight into the curb.
The LAD-Drive Innovation: Structural Disentanglement
LAD-Drive's core philosophy is that the model should decide what to do (Action Decoder) before it figures out how to do it (Diffusion Decoder).
1. The Action Decoder (The "Thinker")
Instead of outputting a single action, LAD-Drive predicts a probabilistic meta-action distribution. This preserves the "belief state" of the agent. If the agent is unsure whether to lane-change or follow the lane, the diffusion planner receives this uncertainty rather than a forced, potentially wrong, choice.
2. The Semantic Bottleneck
A critical discovery in this paper: Raw LLM hidden states (4096-dim) are too noisy for planning. The authors found that bypassing the feature bottleneck caused a 29% drop in performance. By compressing these features to a 512-dim latent space, they filter out linguistic "chatter" and retain pure navigational context.
3. The Action-Aware Diffusion Decoder (The "Doer")
Rather than generating a path from pure Gaussian noise, LAD-Drive uses learned anchors (clustered from expert data). It then uses two steps of truncated diffusion to warp these anchors into the final path.
Fig 1: The LAD-Drive framework. Note the clear separation between the Language module, Action decoder (green), and Diffusion decoder (orange).
Experimental Results: SOTA Performance
LAD-Drive was tested on the LangAuto benchmark in CARLA, which requires agents to follow natural language instructions (e.g., "At the next intersection, take a left and then park").
| Method | Driving Score (Mean) | Route Completion | Infraction Score |
|---|---|---|---|
| LMDrive (Baseline) | 42.9 | 54.8 | 0.80 |
| VLDrive (ICCV '25) | 64.4 | 72.7 | 0.88 |
| LAD-Drive (Ours) | 68.2 | 74.5 | 0.90 |
The "Infraction Score" is where LAD-Drive truly shines. By being "action-aware," the model reduces collisions with vehicles and pedestrians by up to 80% compared to the baseline.
Fig 2: Quantitative ablation showing the drastic reduction in Route Deviation (RD) and Collisions (CV/CP).
Critical Insight: Why Diffusion?
The ablation study (Table III in the paper) reveals a fascinating synergy. Using a Diffusion Decoder alone actually degrades performance compared to a tuned regression baseline. Why? Because generative models are inherently unstable.
The breakthrough comes from Dual Conditioning: Combining high-level Lateral Action guidance (resolving semantic ambiguity) with low-level Ego-Status grounding (ensuring physical feasibility). Without both, the diffusion process lacks the "rails" needed for safe driving.
Conclusion & Future Work
LAD-Drive proves that we don't need a single mammoth head to handle everything. By structurally separating intent from planning and using diffusion as a "refiner," we get the best of both worlds: LLM reasoning and robotic precision.
Limitations: The model still relies on a 2D BEV representation; future iterations might benefit from 4D occupancy or world model forecasting to better handle high-speed dynamic environments.
Keep an eye on this space for the public code release at github.com/iis-esslingen/lad-drive.