DSCache: Solving the Information Decay and Position Overflow in Streaming Video MLLMs

TL;DR

Streaming video understanding requires models to process unbounded data with finite memory. Traditional methods suffer from a "cumulative effect" where old data pollutes new KV caches. Decoupled Streaming Cache (DSCache) solves this by isolating recent "instant" caches from historical "cumulative" caches. It also introduces a position-agnostic encoding strategy to prevent position ID overflow, enabling offline models to work on infinite streams without retraining, achieving a +2.5% accuracy boost on SOTA benchmarks.

Problem & Motivation: The "Cumulative" Trap

In streaming video applications—like autonomous driving or robotics—models cannot simply re-process every frame. They rely on KV Caches to store past context. However, the authors identified a critical flaw in current "uniform" cache strategies:

1. The Cumulative Effect: When we update a KV cache incrementally, the newly encoded "Key" for the current frame is conditioned on the entire historical cache. If older frames are evicted from the cache but their "residuals" still influence the encoding of new tokens, the new tokens become less informative for the current frame.
2. Position Overflow: Transformers are trained on fixed lengths. In an infinite stream, position IDs eventually exceed the training window, causing the model to break down due to out-of-distribution (OOD) positional embeddings.

Methodology: Decouple to Conquer

DSCache introduces two architectural shifts to handle unbounded video streams:

1. Cumulative vs. Instant Decoupling

Instead of one giant evolving cache, DSCache maintains:

• Feature Buffer: A raw buffer of the most recent input features.
• Instant Cache: Constructed on-demand from the Feature Buffer independently of the past. This ensures that the most recent frames (crucial for "what is happening now?") are not corrupted by historical residuals.
• Cumulative Past Cache: Stores the long-term context, updated only when tokens are evicted from the Feature Buffer.

Inference flow: The Instant Cache captures fine-grained recency, while the Cumulative Past Cache provides long-range context.

2. Position-Agnostic Encoding

To solve the position overflow problem, DSCache stores KV entries without positional transformation. At query time, it reassigns contiguous position IDs (e.g., 0 to N) to the active window. The authors provide a formal proof that for RoPE (Rotary Positional Embedding) based LLMs, this "late-binding" of positions is mathematically equivalent to absolute positional encoding as long as relative distances are preserved.

Experiments & Results: Precision in Real-Time

The authors evaluated DSCache using LLaVA-OneVision and Qwen-2.5-VL across major benchmarks like StreamingBench and OVO-Bench.

• SOTA Performance: DSCache consistently outperforms uniform streaming baselines and prior online VideoLLMs like StreamBridge.
• Recency Focus: It shows significant improvements in tasks requiring high-fidelity perception (Attributes +4.5%, Spatial +6.5%).
• Efficiency: While recomputing the instant cache adds small latency, the overall system maintains a stable memory footprint (around 16-17GB) and processes frames within practical 1 FPS limits.

Comparison across multiple benchmarks: DSCache sets a new bar for open-source streaming models.

Critical Analysis & Conclusion

Takeaway

The core insight is that encoding quality matters as much as cache quantity. By preventing "residual pollution" from the past, DSCache allows pretrained offline models to maintain high "recency focus" without losing historical context.

Limitations

The primary trade-off is computational overhead. Recomputing the instant cache for each query adds latency. While the authors propose an "approximate" version to mitigate this, achieving "true" infinite-length understanding without any recomputation remains a challenge. Additionally, like most cache-eviction strategies, extreme long-range dependency tasks (like counting occurrences over an hour) still face information loss.

Future Work

The flexibility of the Position-Agnostic Encoding opens doors for more complex cache operations, such as dynamic retrieval-augmented generation (RAG) where specific "chunks" of memory are re-inserted into the active window at arbitrary positions.