[CVPR 2025] TrajTok: Redefining Video Tokenization through Learned Object Trajectories

TL;DR

The dominant paradigm of splitting videos into fixed grid patches (patchification) is fundamentally inefficient for long-form video understanding. TrajTok breaks this mold by proposing an end-to-end, differentiable tokenizer that groups pixels into semantic trajectories. By treating an object's path through time as a single (or adaptive) token, it achieves state-of-the-art accuracy on benchmarks like Kinetics-400 (+4.8%) while significantly reducing the computational bottleneck of long-range attention.

Problem & Motivation: The "Patch" Tax

In the Transformer era, we have treated video as a 3D stack of bricks. Whether it's a 2-second clip or a 2-minute sequence, models like ViT-3D force a uniform grid of tokens across every frame. This leads to:

1. Redundancy: A static background pixel consumes as much "attention budget" as a moving athlete.
2. Rigidity: Fixed patches cannot adapt to the semantic complexity of the scene.
3. The SAM Bottleneck: Recent attempts to use object trajectories (like the original TrajViT) required external, heavy segmentation models (SAM/SAM2), which are task-agnostic and extremely slow for real-time inference.

The Insight: We don't need "pixel-perfect" masks for video understanding; we need "semantic grouping." TrajTok aims to learn these groups jointly with the downstream task, allowing the model to decide what constitutes a "token" based on what it needs to recognize.

Methodology: The TrajTok Architecture

TrajTok consists of two tightly coupled, differentiable modules that replace the traditional patch-embedding layer.

1. The Universal Segmenter

Instead of per-frame segmentation, TrajTok uses a pool of 128 learnable queries that look at the entire video at once. Through a Perceiver-based cross-attention mechanism, these queries perform "implicit clustering."

• Single Forward Pass: It processes the whole spatiotemporal volume in one go.
• Soft Segmentation: It produces soft similarity maps, ensuring that the entire pipeline remains differentiable so the "reasoning" part of the model can tell the "tokenizer" part what to focus on.

2. The Trajectory Encoder & Adaptive Tokens

Not all trajectories are equal. A background wall needs one token; a person performing a backflip might need four.

• Matryoshka Mechanism: Inspired by Matryoshka Representation Learning, TrajTok can emit $n\in \{1,2,4\}$ tokens per trajectory.
• Diversity Encouragement: Using Fourier positional embeddings, it forces separate tokens within the same trajectory to attend to different temporal or structural aspects, preventing informational collapse.

Experiments: Breaking SOTA and Scaling

The authors evaluated TrajTok in three configurations: TrajViT2 (Encoder from scratch), TrajAdapter (Probing head), and TrajVLM (LLM connector).

Superior Scaling Behavior

A critical finding is that TrajTok scales better with data than previous methods. Because the segmenter is co-trained with the CLIP objective, it "reshapes" its segmentation granularity to favor foreground objects as it sees more data, whereas fixed heuristic methods hit a performance ceiling.

Efficiency in Long Video

In Video-Language Models (VLMs), TrajTok serves as a "semantic connector." In benchmarks like LongVideoBench, TrajVLM outperformed standard patch-pooling by +8.8%, proving that trajectory tokens are a superior "language" for LLMs to understand temporal dynamics.

Critical Insight: Quality vs. Utility

One of the most provocative findings in the paper is that Dice Loss is more important than Cross-Entropy for the segmenter. The authors show that even when the segmentation masks look "coarse" or "imprecise" to the human eye, they provide a much stronger signal for video retrieval and classification than precise but task-agnostic masks. This suggests that the future of computer vision tokenization lies in "Utility-driven" rather than "Fidelity-driven" abstraction.

Conclusion & Future Work

TrajTok successfully decouples video length from token count without the overhead of external tracking models.

• Takeaway: Object trajectories are the "natural" units of video.
• Limitation: The current segmenter can still miss very small, fast-moving objects in complex cluttered scenes due to its low-resolution bottleneck (1/4 scale).
• 展望: Future iterations could likely incorporate "Motion Priors" to further refine token allocation for high-speed action recognition.