[ICRA 2025] NOVA: Revolutionizing 3D Tracking with Next-Step Open-Vocabulary Autoregression

Executive Summary

In the world of autonomous driving, "knowing what you don't know" is a matter of safety. Traditional 3D Multi-Object Tracking (3D MOT) systems often fail when they encounter "Novel" objects—categories not present in their training data. NOVA (Next-step Open-Vocabulary Autoregression) breaks this cycle by transforming tracking from a fragmented geometric matching problem into a generative spatio-temporal reasoning task. By treating a trajectory as a "sentence" and using a 0.5B LLM to predict the "next token" (the match), NOVA achieves a staggering 20% absolute gain in tracking novel objects on the nuScenes benchmark.

Problem & Motivation: The Semantic-Blindness of Heuristics

Current 3D MOT pipelines are typically "closed-set." If a detector hasn't seen a "Tricyclist" during training, it either ignores it or misclassifies it, leading to fragmented trajectories (ID switches) or complete tracking loss. Even recent "Open-Vocabulary" (OV) attempts like Open3DTrack are often decoupled: they use a detector for proposals and then fall back on handcrafted distance-based matching rules.

The authors argue that the core challenge of OV-3D-MOT is maintaining logical consistency under uncertainty. When geometry is jittery and labels are ambiguous, we need a system capable of "common-sense reasoning"—much like how an LLM understands the context of a sentence.

Methodology: Tracking as Next-Token Prediction

Instead of calculating a distance matrix and solving a Hungarian match based on raw numbers, NOVA serializes the trajectory history and the candidate detection into a prompt.

Fig. 1: The NOVA Pipeline—from Open-Vocabulary 3D Detection to Autoregressive Association.

1. The Geometry Encoder & <box> Tokens

LLMs are notoriously bad at processing raw numeric strings (e.g., "x=1.234, y=5.678"). NOVA introduces a Geometry Encoder that maps 3D box coordinates into a continuous embedding space. This embedding is injected into the LLM as a special <box> token, preserving high-precision spatial information. To make this representation robust, they add an auxiliary IoU-quality head that trains the encoder to predict how "good" a box is, providing a signal beyond raw detector confidence.

2. Hybrid Prompting

To prevent the model from over-relying on known category names (like "Car"), the authors use Hybrid Prompting. During training, novel categories are labeled as "Unknown." This forces the LLM to learn the "grammar" of motion and physical size rather than just memorizing labels.

3. Hard Negative Mining

Association errors usually happen between two objects that are very close to each other. NOVA specifically samples these "hard negatives" during training, forcing the model to learn fine-grained discrimination in crowded traffic scenes.

Experiments: Breaking Records with Small Models

The most surprising result is that bigger is not always better. The authors tested LLMs ranging from 0.5B to 3.8B parameters. The Qwen2.5-0.5B model emerged as the winner, offering the best balance of high recall and inference speed (3.4 FPS).

Table 1: NOVA vs. Open3DTrack. Note the massive jump in Novel category AMOTA (2.20 -> 22.41).

Qualitative Edge

Where traditional trackers suffer "ID Switches" (losing track of which car is which) due to geometric jitter, NOVA remains stable. Its generative nature allows it to "fill in the blanks" of a trajectory using temporal context, much like how we predict the next word in a sentence.

Fig. 2: NOVA successfully avoids the Class and ID switches that plague previous SOTA methods in dense urban scenes.

Critical Analysis & Conclusion

NOVA proves that Large Language Models possess an inherent "spatio-temporal logic" that can be unlocked for robotics. By framing tracking as a sequence generation task, the researchers have bypassed the need for complex, handcrafted cost functions.

Limitations:

• Inference Speed: At 3.4 FPS, it is not yet "real-time" for high-speed autonomous driving (usually requiring 10+ FPS).
• Occlusion: The model currently lacks visual/appearance features, meaning it relies purely on geometry and semantics to "guess" through long occlusions.

Takeaway: The "Generative Tracker" paradigm is here to stay. NOVA is a powerful first step toward end-to-end driving systems that can reason about the unknown world as fluently as they process language.