[CVPR 2026] OVGGT: Achieving O(1) Constant-Cost for Infinite 3D Streaming

TL;DR

3D reconstruction from video streams usually hits a "memory wall" where GPU VRAM explodes as the video gets longer. OVGGT breaks this wall. By selectively caching tokens based on FFN residuals and protecting "geometric anchors," it enables high-fidelity 3D reconstruction for arbitrarily long videos (tested up to 10,000 frames) using constant memory and compute—all without any extra training.

The Bottleneck: The KV Cache is a Growing Burden

In the era of Geometric Foundation Models (like DUSt3R or VGGT), we’ve moved from complex multi-stage pipelines to end-to-end Transformer inference. However, streaming variants like StreamVGGT utilize a Causal KV Cache. While this allows single-pass processing, the cache grows linearly:

• 100 frames $\approx$ 10 GB VRAM.
• 300 frames $\to$ Out of Memory (OOM) on most consumer GPUs.
• Latency increases every step because the Attention mechanism has to look back at an ever-growing history.

The industry needed a way to prune this cache without losing the "spatial memory" required to keep 3D points aligned.

Methodology: Intelligence via Residuals and Anchors

The researchers introduced two key innovations that make OVGGT an $O(1)$ operator.

1. Self-Selective Caching (SSC): FFN as a Saliency Map

How do you decide which tokens to throw away? Standard methods often rely on Attention Weights, but those are hidden inside the FlashAttention kernel for efficiency.

OVGGT's insight: FFN residuals ($\Vert {\lambda }_2\cdot FFN(LN(h)){\Vert }_2$) naturally highlight geometric saliency.

• Shallow layers focus on texture.
• Mid-layers focus on geometric structure (edges/corners).
• Deep layers focus on semantic boundaries.

By using these values, OVGGT performs "Activation Smoothing" to keep spatially coherent groups of tokens, ensuring the depth heads don't see a "scattered" and noisy history.

2. Dynamic Anchor Protection (DAP): Guarding the Origin

If you evict tokens purely by "importance," you might accidentally delete the tokens that define your world coordinate system. OVGGT prevents this via:

• Global Initial Anchor: Permanently protecting the first frame to keep the origin stable.
• Historical Anchors: Dynamically registering new "landmarks" when the camera moves to a new area (low view overlap), providing a "long-range" reference tether.

Experimental Results: Better Accuracy, Zero Growth

OVGGT was tested on everything from indoor rooms (7-Scenes) to 10,000-frame "Ultra-Long" environments.

• Stability: Unlike StreamVGGT, which accumulates noise from redundant tokens until it crashes, OVGGT maintains a flat error rate.
• Efficiency: It is the only causal method that maintains a constant VRAM reserved profile and stable FPS regardless of sequence length.

The graph above shows OVGGT (blue) maintaining 10GB VRAM and ~5 FPS indefinitely, while competitors either explode in memory (red) or run significantly slower (green/yellow).

Critical Analysis: Why This Matters

The most striking result is that OVGGT actually outperforms the full-cache baseline in many scenarios. This suggests that in 3D geometry, redundant history isn't just a memory problem—it's a noise problem. By pruning the cache, the model focuses its attention on the most geometrically relevant features.

Limitations: As a single-pass causal pipeline, it cannot "back-correct" errors. If the camera tracking drifts, the error is baked into the map. The authors suggest "staged streaming" (periodic global refinement) as the next frontier.

Conclusion

OVGGT proves that we can achieve "infinite" 3D perception on consumer hardware. By treating the KV cache as a dynamic, selectively managed memory rather than an unbounded dumping ground, it sets a new standard for real-time spatial AI applications.