123D: Building the "Common Crawl" of Autonomous Driving

TL;DR

Autonomous driving research is currently stuck in "dataset silos" where models trained on nuScenes fail on Waymo due to minor differences in sensor rigs or label formats. 123D is a groundbreaking open-source framework that unifies 9 flagship datasets (3,300+ hours of data) into a single, standardized API. It solves the nightmare of data synchronization and coordinate alignment, enabling the community to finally train "generalist" driving models at scale.

The "Babel" Problem in AV Data

In the world of Large Language Models (LLMs), researchers have Common Crawl. In General Robotics, we have Open X-Embodiment. But in Autonomous Driving (AD), every new dataset release feels like building a new tower from scratch.

Current AD datasets are fragmented by:

• Modalities: Varying numbers of cameras and lidars with different FOVs.
• Synchronization: Some datasets provide 10Hz "keyframes," others provide raw asynchronous event streams.
• Conventions: Is "X" forward or right? Is the origin at the rear axle or the IMU?
• Dependencies: Outdated C++ libraries required to read proprietary file formats (e.g., TFRecord vs. .json).

This lack of infrastructure prevents the field from achieving the "Scaling Laws" seen in NLP.

Methodology: The Stream-Based Architecture

The core innovation of 123D is treating every sensor and annotation not as a fixed frame, but as an independent event stream.

1. Unified Log Format

Leveraging Apache Arrow IPC, 123D stores data in a columnar format that allows for high-speed I/O. Instead of forcing a global frame rate, it stores each modality with its native timestamp. A Sync Table then acts as a look-up index to align these streams on-the-fly.

2. Standardized Geometry

The framework enforces:

• Body Frame (ISO 8855): X-Forward, Y-Left, Z-Up.
• Camera Frame (OpenCV): X-Right, Y-Down, Z-Forward.

This rigorous alignment means a researcher can write one dataloader and immediately train on nuScenes, Waymo (WOD), and Argoverse 2 simultaneously.

Figure 1: The 123D Pipeline—from fragmented raw data to a unified Apache Arrow format and Scene API.

Experiments: The High Cost of Specialization

The authors conducted a massive cross-dataset study that exposes the hidden biases in our current SOTA models.

3D Object Detection: The "Rig Gap"

When training a BEVFormer on nuScenes and testing it on Waymo, there is a massive performance drop (e.g., from ~0.50 NDS to ~0.20 NDS). Interestingly, the authors found that Joint Training (Mixed-5)—training on five datasets at once—almost entirely closes this gap.

Key Insight: Architectural choice matters. Models with explicit Bird’s-Eye-View (BEV) grid representations generalized better than those using purely coordinate-based embeddings (like PETR), suggesting that spatial inductive biases help bridge the gap between different sensor configurations.

Figure 2: Performance (NDS) of 3D detectors. Mixed-dataset training (dashed lines) consistently outperforms or matches single-dataset specialization across the board.

Generalization in Planning

Using PufferDrive (a high-speed RL simulator), the authors proved that Reinforcement Learning agents also suffer from "domain shock." A planner trained only on nuPlan struggled on Waymo maps. However, agents trained on the Mixed-3 dataset showed superior zero-shot transfer even to synthetic CARLA environments.

Critical Analysis & Conclusion

123D is a much-needed "engineering-first" contribution. By removing the friction of data curation, it shifts the focus from "how do I parse this data?" to "how do I build a model that understands the world?".

Limitations to Watch:

• Web-Scale Streaming: While it uses Arrow, it doesn't yet natively support streaming from S3 buckets at the scale of billions of frames.
• Semantic Alignment: While coordinates are unified, "labels" are still tricky. A "truck" in one dataset might be a "large vehicle" in another. Automated label alignment using LLMs is the likely next frontier for 123D.

Takeaway: If you are working on Autonomous Driving, stop writing custom parsers. 123D is now the standard starting point for scaling your multi-modal models.