LookWhen: Breaking the Video Redundancy Barrier with Smart Selection

TL;DR

Video recognition is notoriously expensive because Transformers traditionally treat every pixel-patch with equal importance. LookWhen changes this by learning exactly when, where, and what to compute. By using a lightweight "Selector" to pick only the most unique 5-30% of patches for a "Deep Extractor" to process, it achieves a staggering 6.7× throughput increase over state-of-the-art models like InternVideo2 while maintaining the same accuracy.

The Problem: The High Cost of Video Redundancy

In a video of a person throwing a ball, most background patches remain static or predictable across frames. Processing every single token in a 16-frame clip at 224 resolution is computationally wasteful.

Existing solutions like Token Merging (ToMe) or Pruning attempt to fix this mid-stream. However, they usually:

1. Process everything initially: They still pay the full price for the first several layers.
2. Are Hardware-unfriendly: Complex clustering or masking often breaks optimized kernels like FlashAttention, meaning theoretical FLOP savings don't translate to faster real-world speeds.

Methodology: The Three Pillars of LookWhen

LookWhen factorizes the problem into two distinct components:

1. The Shallow Selector (Efficiency First)

The Selector sees the whole video but at a halved resolution and with only 3 transformer layers. Its job is to generate a "Selector Map" that ranks every patch's importance.

2. The Top1-Distance Metric (The "When" and "Where")

How do you train a model to know what is important? Instead of relying on noisy attention maps (which often suffer from artifacts in video models), LookWhen introduces Top1-Distance.

• It measures a patch's distance to its nearest neighbor in the feature space of a teacher model (DINOv3).
• Intuition: If a patch is very different from all others, it represents a unique event or object—this is what the Extractor needs to see.

3. Dual-Teacher Distillation (The "What")

The Extractor only sees the "Top-K" patches but must represent the entire video. To achieve this, it distills knowledge from:

• InternVideo2: For high-level, global video semantic understanding.
• DINOv3 (Time-Normalized): By normalizing frame-wise features, the model is forced to focus on what changes within the video, which is critical for fine-grained action recognition (e.g., "folding" vs. "unfolding" a shirt).

Performance: Actual Speed Matters

The true highlight of LookWhen is its Pareto-dominance in real-world benchmarks. While many papers claim efficiency based on FLOPs (Floating Point Operations), LookWhen delivers on Throughput (videos per second).

• Accuracy vs. Compute: On Kinetics-400, LookWhen (70% sparse) hits 84.6% Top-1, outperforming the dense InternVideo2-B baseline despite being significantly faster.
• Throughput Advantage: Because LookWhen uses standard, highly-optimized operations and simply processes fewer tokens, it avoids the overhead of complex merging algorithms.

Critical Analysis: Why This Matters

LookWhen’s success hinges on the Inductive Bias that video is fundamentally sparse. By moving the "Selection" to the very beginning of the pipeline (the shallow selector), the model avoids the "peak activation" memory bottleneck that plagues other adaptive models.

Limitations: Currently, the model relies on a strong teacher (like DINOv3 or InternVideo2) during pre-training. If the teacher has biases or fails to ground features spatially (positional misalignment), the selector's performance will degrade. However, as "foundation models" become more robust, this dependency becomes a strength.

Conclusion

LookWhen is a masterclass in combining Academic Rigor with Practical Utility. It proves that we don't need to process the whole video to understand it—we just need to know where to look. For researchers and engineers working on large-scale video processing, this framework offers a clear path toward sustainable, high-speed AI.

Key Insight: Don't just compute less; compute smarter by measuring uniqueness.