[CVPR 2025] LFG: Learning to Drive is a Free Gift — Scalable Autonomy via 4D Video Pretraining
Learning to Drive is a Free Gift: Large-Scale Label-Free Autonomy Pretraining from Unposed In-The-Wild Videos
LFG (Learning to drive is a Free Gift) is a label-free, teacher-guided pretraining framework that learns unified pseudo-4D representations (geometry, semantics, motion, and future evolution) from unposed, in-the-wild driving videos. It achieves state-of-the-art results on the NAVSIM planning benchmark using only a single front-camera view, outperforming many multi-view and LiDAR-based methods.
TL;DR
LFG is a breakthrough in label-free pretraining for autonomous driving. By leveraging "unposed" (no camera extrinsic data) videos from the internet (e.g., YouTube), LFG learns to reconstruct 3D geometry, semantics, and dynamic motion. It treats future prediction as a next-token prediction problem in the latent space. Remarkably, using only a single front-facing camera, LFG outperforms established benchmarks like UniAD and Hydra-MDP that utilize multi-camera and LiDAR suites.
Problem & Motivation: The "Label Bottleneck"
The current paradigm in AD (Autonomous Driving) relies heavily on high-fidelity labels: LiDAR point clouds, 3D bounding boxes, and expert trajectories. This creates a data bottleneck. While internet-scale video datasets like OpenDV exist, they lack the structural "pose" information required by classical SfM (Structure from Motion) pipelines.
The authors argue that if we want "GPT-scale" scaling for driving, we must move beyond static frame consistency. A driving model doesn't just need to see the now; it must anticipate the future 3D state of both the ego-vehicle and the dynamic agents surrounding it.
Methodology: Distilling the 4D World
LFG transforms the (Permutation-equivariant visual geometry learning) architecture into a temporal predictive machine.
1. The Architecture
LFG uses a DINOv2-based encoder to extract features from observed frames. These are passed through a Causal Autoregressive Transformer, which generates future tokens. These tokens are then decoded into:
- 3D Point Maps: Dense geometry per pixel.
- Camera Poses: 6-DoF ego-motion.
- Semantic Layouts: Class probabilities (Road, Vehicle, etc.).
- Motion Masks: Identifying dynamic vs. static pixels.
2. Teacher-Guided Supervision
Since the source videos are unlabeled, LFG uses a "council of teachers":
- Geometry Teacher: A model with access to the entire sequence (including future frames the student hasn't seen).
- Semantic Teacher: SegFormer (Cityscapes-pretrained).
- Motion Teacher: A pipeline involving Grounded SAM2 and CoTracker3 to identify and track moving objects, converting 2D tracks into 3D displacements via the point maps.
Experiments & SOTA Performance
LFG was evaluated on the NAVSIM planning benchmark. This is a non-reactive simulator where the model must predict safe trajectories.
Planning Breakthrough
LFG sets a new standard for single-camera planning. With 100% data, it achieves a 85.2 PDMS, outperforming UniAD (which uses 6 cameras). Even more impressive is its sample efficiency: at 10% data, LFG (81.4) matches the performance of a DINOv3-based planner using 100% data.
Predicting the Future
The model's ability to "hallucinate" the future is visible in its semantic segmentation results. Even when LFG can no longer "see" (predicting frame 6 from frame 3), it correctly evolves the scene, outperforming the SegFormer teacher which actually had the RGB input for those frames.
Deep Insight: Why "Free Gift"?
The title "Learning to Drive is a Free Gift" refers to the fact that the geometric and semantic priors necessary for driving are already embedded in the temporal flow of YouTube videos.
- Inductive Bias: By forcing the model to predict future point maps and ego-poses, it naturally learns the physics of motion and the constraints of 3D space.
- Unified Encoding: Instead of separate backbones for perception and planning, the "autonomy tokens" produced by LFG are inherently carry downstream task-relevant information.
Limitations & Future Work
While LFG is powerful, its future horizon is currently short (approx. 3-6 frames/0.6 to 1.2 seconds if 5Hz). Extending this to long-range reasoning (5-10 seconds) remains a challenge for autoregressive latent models due to compounding errors. Furthermore, the model currently relies on a single camera; moving to a multi-view "internet video" training setup as more multi-view datasets appear will likely be the next frontier.
Conclusion
LFG proves that we don't need expensive LiDAR labels to build a world-class driving foundation model. By scaling up label-free pretraining on in-the-wild videos, we can "gift" our models a deep understanding of geometry and semantics, providing a highly efficient base for any downstream autonomy task.