[arXiv 2026] ReCoSplat: Mastering Long-Sequence Autoregressive Gaussian Splatting

TL;DR

ReCoSplat is a breakthrough in online 3D Gaussian Splatting. Unlike previous "one-shot" or offline models, it processes a continuous stream of images (unposed or posed) and builds a consistent 3D world in real-time. By using a "Render-and-Compare" mechanism, it fixes the common failure mode where predicted camera poses don't match training distributions. Furthermore, it manages to reconstruct scenes with hundreds of images on consumer hardware through a highly efficient KV-cache pruning strategy.

The "Assembly Pose" Dilemma

In feed-forward 3DGS, the model predicts local Gaussians in camera space and then "assembles" them into a global world using camera poses. This creates a catch-22 for autoregressive models:

1. Train with Ground-Truth (GT) Poses: The model becomes "spoiled." At inference, when it must use imperfect predicted poses, the local Gaussians become misaligned.
2. Train with Predicted Poses: The training becomes unstable because GAussian prediction is coupled with noisy pose estimation.

ReCoSplat solves this by saying: Let the model see its own mistakes.

Methodology: Perception as Analysis-by-Synthesis

The core of ReCoSplat is the Render-and-Compare (ReCo) module. When a new image arrives, the model renders the previously accumulated scene from the expected new viewpoint.

The model then compares this rendering with the actual incoming observation. Any visual or geometric discrepancy acts as a "conditioning signal" (via cross-attention) that tells the Gaussian prediction head exactly how to adjust the new primitives to fit the existing scene. This effectively bridges the gap between GT and predicted pose distributions.

Breaking the Memory Wall: 10x KV Cache Compression

Standard Transformers suffer from $O(N^2)$ or $O(N)$ complexity that quickly exhausts VRAM in long sequences. ReCoSplat introduces two brutal but effective optimizations:

• Early Layer Truncation: The authors observed that the first 10 global attention layers focus on local features. They simply stop storing KV caches for these layers.
• Selective Retention: For the remaining layers, they only keep the KV tokens of one "representative" view per chunk of 8 images, using a Prompt Register Token to mark these anchor views.

Experimental Performance

ReCoSplat was tested on massive datasets like DL3DV and ScanNet++. In the unposed (pose-free) setting, it significantly outclasses previous autoregressive attempts (like S3PO-GS) and approaches the quality of offline models that see the whole sequence at once.

The results show sharper geometry and fewer "floaters" or artifacts compared to baselines. Even on out-of-distribution data (ACID, RealEstate10K), the method shows robust generalization, proving that the ReCo module isn't just memorizing training data—it's learning the geometric relationship between rendering and observation.

Critical Insight & Conclusion

The industry is moving from "reconstruct-then-render" (Offline) to "reconstruct-while-moving" (Autoregressive). ReCoSplat's biggest contribution is proving that explicit feedback loops (rendering the current state) are more powerful for stability than complex training curricula or data augmentation.

Limitations: While robust, ReCoSplat still depends on a decent initial pose estimate. If the pose error is catastrophic (e.g., a total lack of texture for matching), the ReCo module's signal becomes too noisy to recover the geometry perfectly. Future work integrating tighter SLAM-loop closures into this architecture would likely eliminate these remaining edge cases.

Takeaway: If you want a transformer to handle long-term 3D memory, don't store everything. Prune the early layers and use "visual feedback" to correct for the noise of the real world.