[CVPR 2025] Proact-VL: Shaping VideoLLMs into Proactive Real-Time AI Companions

TL;DR

Proact-VL is a general framework that transforms Video Large Language Models into proactive agents. By introducing a "Decide-then-Generate" mechanism and a massive "Live Gaming Dataset," it masters the art of gaming commentary and player guidance. It doesn't just watch; it knows exactly when to speak and when to stay silent, matching human-like interaction speeds.

Motivation: The "Chatterbox" vs. "Latency" Dilemma

In the realm of AI companions (like game commentators or live-stream guides), current models fall into two traps:

1. Offline "Thinkers": Models like GPT-4o provide high-quality analysis but usually wait for a full video clip to finish, resulting in "coarse" timing.
2. Streaming "Chatterboxes": Real-time models often produce a constant stream of text, which becomes overwhelming and unnatural for a user looking for a companion, not a narrator.

The authors observed that human-like companionship requires precise control over Timing, Quality, and Quantity.

Methodology: The "Decide-then-Generate" Pipeline

1. Chunk-wise Input Schema

Proact-VL processes video in 1-second "chunks." Each step, the model receives a triplet: $(V_t,Q_t,B_t)$ representing current visuals, optional user queries, and environmental context (what other commentators just said). This ensures the model is always "in the moment."

2. The <|FLAG|> Token: The Decision Anchor

Instead of forcing the LLM to predict a "SILENCE" word (which is hard to calibrate), the authors inserted a semantic-free <|FLAG|> token.

• The hidden state of this token is fed into a lightweight MLP head.
• This head outputs a probability score $p_t$.
• If $p_t>au$, the model triggers text generation.

Figure: The Proact-VL workflow. Multi-source tokens flow through the transformer, where a dedicated response head makes the binary decision to speak or remain silent.

3. Training for Stability

Teaching a model to be silent is harder than teaching it to speak. Proact-VL uses:

• Transition-Smoothed Loss: Penalizes the model more for missing the start or end of a speech segment.
• Stability Regularization: Ensures the model doesn't "jitter" (rapidly switching between speaking and silence) and keeps its overall speaking rate close to a human baseline.

Experiments: Dominating the Gaming World

The authors built the Live Gaming Benchmark, covering titles like Elden Ring, League of Legends, and Minecraft.

SOTA Performance

Proact-VL outperformed commercial juggernauts like GPT-4o and Gemini 2.5 Pro in real-time settings. While offline models are "smarter," they can't match Proact-VL's TimeDiff (temporal accuracy) and F1 (proactivity score).

Table: Proact-VL significantly reduces TimeDiff (lower is better) and increases F1 scores across Solo, Co-Commentary, and Guidance tasks.

Real-Time Efficiency

Because the model uses a Dual-Cache Sliding Window (with a clever Reverse-RoPE correction to prevent positional drift), it can run indefinitely on 10-15 FPS streams without running out of memory or losing track of the "system prompt" instructions.

Deep Insight: Social Intelligence in Co-Commentary

One of the most impressive feats of Proact-VL is Co-commentary etiquette. In tests, the model learned to "listen" to other speakers. If a human co-host started a deep analysis, Proact-VL would withhold its comments until a natural pause, then enter with a confirming observation. This "turn-taking" is a massive leap from standard VLM behaviors.

Critical Analysis & Conclusion

While Proact-VL is a breakthrough, it faces two main hurdles:

• Fine-grained Perception: At 2 FPS, it might miss a "flicker" of a Rare Item in an RPG or a split-second headshot in a shooter.
• Hallucination: Like all LLMs, it can occasionally "see" a 2K gold lead that doesn't exist if the HUD text is too small (OCR failure).

The Takeaway: Proact-VL proves that the future of AI isn't just "Question & Answer"; it's "Observation & Participation." By treating "When to speak" as a first-class citizen in model training, we move one step closer to truly human-like digital companions.