[CVPR 2025] Fail2Drive: Exposing the Brittle Logic of SOTA Autonomous Driving Models

TL;DR

Autonomous driving models have become remarkably good at "gaming" benchmarks by memorizing simulator-specific assets. Fail2Drive introduces the first paired-route benchmark in CARLA to measure the Generalization Gap. By swapping standard pedestrians for animals or placing construction cones in irrelevant lanes, the authors reveal that even the best models (including Vision-Language and LiDAR-based ones) suffer an average success rate drop of 22.8%, often failing to grasp the basic concept of an "obstacle."

The "Memorization" Problem in Autonomous Driving

Why do models that top the CARLA leaderboard often fail in the real world? The authors of Fail2Drive argue that current evaluation protocols are fundamentally flawed. Most models are tested on the same types of assets and scenarios they saw during training. Consequently, they don't learn to drive; they learn to recognize templates. For example, a model might not learn to "avoid obstacles," but rather "if I see this specific CARLA traffic cone mesh, shift left."

Methodology: The Paired-Route Insight

To move beyond absolute performance scores, Fail2Drive uses a Paired Evaluation design.

1. In-Distribution Route: A standard CARLA scenario (e.g., a pedestrian crossing).
2. Generalization Route: The same location and traffic, but the pedestrian is replaced by an animal or a differently textured object.

This isolates the distribution shift, turning qualitative failures into quantitative diagnostics. The benchmark covers four critical pillars: Robustness, Visual-Lateral, Visual-Longitudinal, and Behavioral generalization.

Where Models Break: Key Findings

The study evaluated seven heavyweight models, including UniAD, TransFuser++ (TF++), and the VLM-based SimLingo.

1. The LiDAR Trap

One might think LiDAR-based models are immune to visual deception. Fail2Drive proves otherwise. In the "Wall" scenario—where a wall with a printed road image blocks the path—TF++ (LiDAR+Camera) failed to stop, colliding at full speed. This suggests the model over-relies on visual cues to "validate" LiDAR points, essentially ignoring the physical obstacle because it didn't look like a CARLA-specific vehicle.

2. Failure of VLM Generalization

Vision-Language Models like SimLingo were expected to generalize better due to their large-scale pretraining. However, SimLingo showed a massive -23.1% HM drop. In scenarios with pedestrians walking on the road, it often hallucinated "cars" or "cyclists" in its language output, failing to identify the pedestrians and causing collisions in 87% of cases.

3. Overfitting to Symbols

Privileged models like PlanT 2.0 showed extreme sensitivity to symbolic cues. When a construction warning sign was removed — even if the obstacle remained visible — performance collapsed from 100 to 0. It seems these models learn "Sign -> Change Lane" rather than "Occupied Space -> Avoid."

The Fail2Drive Toolbox

To support the community, the authors released a toolbox to extend CARLA with:

• 17 Animal Assets with full animations.
• Deceptive Assets: Image walls and obscured stop signs.
• PDMLite-F2D: An improved expert policy for baseline comparisons.

Critical Analysis & Conclusion

Fail2Drive is a "wake-up call" for the end-to-end driving community. It demonstrates that scale (more data) is not enough if the data lacks structural diversity. The fact that models fail to perceive "unseen" assets as physical obstacles suggests that our current "Representation Learning" is too tightly coupled to the training distribution's texture and geometry.

Future Outlook: The industry must move toward models that possess a more fundamental physical intuition of the world (e.g., World Models or Occupancy-based planners) rather than high-level semantic template matching.