Tracing the Arrow of Time: Why Your Video-LLM Can't Tell Past from Future

Executive Summary

TL;DR: Most Video-LLMs fail the "Arrow of Time" (AoT) test—the ability to tell if a video is playing forward or backward. This paper reveals that the problem isn't necessarily the vision encoder, but the projector (like Q-Former) which acts as a bottleneck, destroying temporal cues before they reach the LLM. By switching to a time-preserved MLP projector and adding AoT supervision, the researchers reached 98.1% accuracy, effectively "out-clocking" humans.

Positioning: This work is a crucial diagnostic study that shifts the focus from "scaling up data" to "architectural info-flow," providing a blueprint for fixing temporal blindness in multimodal systems.

The "Arrow of Time" Problem

Humans have an intuitive grasp of entropy and causality. We know a shattered glass doesn't reform and fly back onto a table. This "Arrow of Time" is a fundamental pillar of physical reasoning. However, even frontier models like GPT-4o struggle to identify if a clip is reversed.

The authors investigated whether the issue lies in the Vision Encoder (the eyes) or the Projector/LLM (the brain).

Methodology: Tracing the Flow

The researchers categorized encoders into two camps:

1. Frame-centric: (e.g., Qwen2-VL, DINOv2) These treat videos as stacks of images.
2. Video-centric: (e.g., InternVideo2, V-JEPA) These use "tubelets" or explicit cross-frame modeling.

1. The Encoder Insight

Probing experiments showed that video-centric encoders do know which way time flows (reaching ~88% accuracy). Frame-centric encoders, however, are essentially "time-blind."

2. The Projector Bottleneck

This was the "aha!" moment. Even when using a strong video-centric encoder, models like InternVideo2VL failed on AoT. Why? The Q-Former projector. By using learnable queries to compress video features into a small token set, the Q-Former effectively "scrambles" the temporal sequence.

Figure 1: The MLP projector preserves temporal sensitivity, while the Q-Former disrupts it.

The authors proposed a Time-Preserved MLP:

• How it works: It pools spatial dimensions ($HimesW$) but keeps the temporal dimension ($T$) intact.
• Result: The LLM finally "sees" the sequence of events in order.

Experiments: Surpassing Human Accuracy

By combining a video-centric encoder (InternVideo2-Stage1) with a time-preserved projector, the model didn't just improve—it dominated.

Table 1: Comparison across temporal benchmarks. Note the massive jump (+47.4 points) in AoT performance.

Key Breakthroughs:

• Beyond Humans: Human accuracy on the AoTP benchmark is ~89%. This model hit 98.1%.
• Generalization: Fixing the AoT bottleneck also improved general temporal tasks like "Action Sequence" and "Moving Direction" (up to +6.0 points).
• The Stage-1 Paradox: The authors found that "CLIP-style" language alignment (Stage 2) actually hurts temporal reasoning by making representations more implicit/language-focused rather than physically grounded.

Critical Analysis & Future Outlook

Takeaway: If your goal is physical world understanding (robotics, surveillance, etc.), stop using Q-Formers or heavy temporal compression. Preserve the "Time Arrow."

Limitations:

• Frame Sparsity: The model still uses only 16 frames. In complex tasks where the evidence of time-reversal is subtle (like a person blinking), 16 frames might miss the cue entirely.
• Implicit vs. Explicit: While the model is better at "what happened when," it still doesn't possess a formal "world model" of physics; it is a superior pattern recognizer of temporal flow.

Future Work: This research paves the way for "Temporally-Aware Instruction Tuning," where models are explicitly taught the laws of physics and causality through time-reversal tasks, making them far more reliable for real-world interaction.