[CVPR 2025] RIVER: Bridging the Gap Between Offline Perception and Real-Time Interaction

TL;DR

The AI community has mastered "watching" videos, but can it "interact" with them in real-time? RIVER Bench is a new evaluation framework that moves beyond offline QA to test Video LLMs on their ability to remember the past, perceive the present, and anticipate the future in streaming environments. By introducing a Long-Short Term Memory module, the authors show how even "offline" models can be adapted for live, high-stakes interactions like AR navigation and robotic supervision.

Problem: The "Offline" Bottleneck

Most current Multimodal LLMs (MLLMs) are like film critics: they watch the entire movie and then write a review. In the real world, AI needs to be a co-pilot. Imagine an AR assistant that helps you find your keys; it cannot wait for the "video" to end to tell you where you left them 10 minutes ago.

Prior works suffer from:

1. Memory Overflow: Standard Transformers have a quadratic complexity that fails as video length grows.
2. Temporal Forgetting: Models lose track of early events as new frames arrive.
3. Lack of Proactivity: Models only respond when prompted, rather than alerting users when a specific future condition is met.

Methodology: Thinking in Slots and Windows

To transform static MLLMs into online interactive agents, the authors introduced a dual-layer approach:

1. Long-Short Term Memory Framework

The model doesn't try to remember every pixel. It uses a Sliding Window (1 fps) for immediate context (Short-term) and a Compressed Memory Bank for the past (Long-term). Using a nearest-neighbor averaging strategy, the module abstracts old information into high-level semantic tokens (Memory Slots).

Figure: The pipeline transforming standard MLLMs into online-capable agents using Long-Short Term Memory.

2. The RIVER Bench Tasks

RIVER categorizes the interaction into three temporal domains:

• Retro-Memory: "Where did I put the bag 30 minutes ago?" (Tests forgetting curves).
• Live-Perception: "What is happening right now?" (Tests latency and semantics).
• Pro-Response: "Tell me when the water starts boiling." (Tests future-state prediction).

Experiments: Do Models Actually "Forget"?

The authors evaluated 1,067 videos and over 4,000 questions. A fascinating finding was the "Memory Curve." Unlike the human Ebbinghaus forgetting curve, MLLMs with memory modules exhibit much higher stability within a 1-hour window, suggesting their memory mechanisms—while inspired by humans—are fundamentally more persistent but less "flexible."

Table: Comparison across different recall durations (Short to Very Long).

Key Performance Insights:

• GPT-4o remains the king of live perception, but open-source models like VideoChat-Flash are catching up when augmented with the RIVER training paradigm.
• Causal Reasoning is the Achilles' heel: Most models fail at "Causal Cues" (Why did that happen?), proving that just "seeing" an event isn't enough to understand its logic.
• Proactive Gains: Fine-tuning on RIVER data yielded an 11.28% accuracy boost in timing-sensitive tasks.

Critical Analysis & Conclusion

RIVER Bench is a significant step toward Real-Time AI Agents. It identifies that the current SOTA is good at "Retrospective Memory" but struggles significantly with "Proactive Anticipation."

Limitations: The primary drawback is the lack of audio integration. Real-time interaction is inherently multi-sensory; a beep from a microwave is often more informative than the visual change. The authors have pledged to include audio in future iterations.

Future Outlook: This research suggests that the future of Video LLMs isn't just "more tokens," but smarter compression and retrieval mechanisms. If you are building for AR or Robotics, the "Sliding Window + Memory Slot" architecture is likely the blueprint for the next generation of online MLLMs.

For more technical details, visit the RIVER Bench GitHub.