[CVPR 2025/ArXiv] FILT3R: Latent State Adaptive Kalman Filter for Streaming 3D Reconstruction

Executive Summary

TL;DR: FILT3R transforms the heuristic "update gate" in streaming 3D reconstruction into a principled Adaptive Kalman Filter. By propagating per-token uncertainty, the model learns when to trust its long-term memory and when to adapt to new scene changes, effectively solving the catastrophic drift common in long-horizon inference (1000+ frames).

Background Positioning: This work is a robust "plug-and-play" architectural improvement. It sits atop existing streaming backbones (like CUT3R) and replaces the state update logic with a training-free layer inspired by classical control theory, achieving SOTA stability and length generalization.

Motivation: The Stability Crisis in Streaming

Streaming 3D reconstruction faces a fundamental trade-off: Persistence vs. Plasticity.

1. Uniform Overwrites (e.g., CUT3R): Trust the new frame completely. Result: Rapid forgetting of the past and jittery geometry.
2. Heuristic Gating (e.g., TTT3R): Scale updates based on attention patterns. Result: Heuristic stability that often fails when the sequence length exceeds the training horizon (typically 64 frames).

The authors' core insight is that the latent tokens in a recurrent model are not just data; they represent a stochastic belief state. If we can quantify the uncertainty of this belief, we can mathematically derive the optimal update gain.

Methodology: Kalman Filtering in Token Space

FILT3R treats the recurrent update as a State-Space Model:

• Process Model: $s_t=s_{t-1}+w_t$. The scene evolves (or the camera moves). $Q_t$ (Process Noise) tracks this.
• Measurement Model: $ilde{s}_t=s_t+v_t$. The decoder's output is a "noisy sensor" reading. $R$ (Measurement Noise) represents this.

The Update Mechanism

The core of FILT3R is the Kalman Gain ($k_t$): $k_t=\frac{p_t^{-}}{p_t^{-}+r}$ Where $p_t^{-}$ is the predicted variance. Crucially, as the model stays in a stable scene, the variance $p$ shrinks, the gain $k$ drops, and the memory becomes "conservative." When the camera moves rapidly, the temporal drift (difference between candidate states) spikes, increasing $Q_t$, which re-opens the gain for fast adaptation.

Figure 1: Comparison of update rules. FILT3R (right) propagates uncertainty, enabling time-varying gains that adjust to scene stability.

Experiments & Results

The researchers tested FILT3R across Pose Estimation, Depth, and Full Reconstruction.

1. Long-Horizon 3D Reconstruction

On the 7-Scenes dataset, at length 1000 (vastly beyond the training horizon), FILT3R maintains geometric coherence while other methods fail or run out of memory (OOM).

2. Trajectory Stability

Using Origin-Aligned ATE (ATEorig), which highlights cumulative drift rather than local error, FILT3R shows a dramatic reduction in drift. On TUM-800, it achieved 0.107 compared to TTT3R's 0.214.

Figure 2: Qualitative long-horizon reconstruction. Note the fragmented surfaces in TTT3R (red boxes) versus the coherent geometry produced by FILT3R.

Critical Analysis & Insight

Why does it work? The ablation study reveals that Variance Propagation is the "secret sauce." Without the recursive update of the variance ($p_t$), the gain cannot "settle" into a confident state.

Interestingly, the authors found that making the Measurement Noise ($r$) adaptive (e.g., based on attention) actually hurt performance. This is because "scene change" and "attention novelty" are highly correlated, causing the gain to overshoot and become unstable—proving that a stable, fixed $r$ serves as a necessary "anchor" for the Kalman filter.

Limitations: While FILT3R is training-free, its performance is still bounded by the pre-trained backbone. If the backbone's candidate predictions are biased, the filter will eventually track that bias, though much more slowly than previous methods.

Conclusion: A Return to Fundamentals

FILT3R proves that deep learning researchers don't always need "more data" or "larger models" to solve stability problems. Often, the solution lies in the elegant fusion of modern architectures with the robust mathematical frameworks of the past—in this case, reaching back to 1960 to bring Kalman's insights into the world of 3D latent tokens.