[CVPR 2024] tttLRM: Scaling 3D Reconstruction to Long Contexts via Test-Time Training

TL;DR

Reconstructing 3D scenes from a continuous stream of images has long been a bottleneck for AI. Standard Transformers explode in computational cost as the number of views increases. tttLRM changes the game by using Test-Time Training (TTT). Instead of storing every image in a massive memory cache, it "learns" the scene into the weights of the model itself. The result? Linear complexity, autoregressive streaming, and SOTA 3D Gaussian Splatting from up to 64+ views.

The Scalability Wall in 3D Reconstruction

The community has seen a surge in Large Reconstruction Models (LRMs) that can turn images into 3D objects in a single forward pass. However, these models are usually "short-sighted," limited to 2-4 views because they rely on standard Attention mechanisms where complexity grows quadratically ($O(N^2)$).

When you try to reconstruct a whole room or a city block, you need dozens or hundreds of images. Prior works like Long-LRM pushed this to 32 views, but the bidirectional attention remained a barrier for real-time streaming. We needed a mechanism that functions like human memory: observing a stream, building an abstract internal representation, and decoding it only when needed.

Methodology: Neural Weights as 3D Memory

The core innovation of tttLRM is the use of LaCT (Large Chunk Test-Time Training) blocks.

1. Fast Weights as Latent Space

In tttLRM, the relationship between input images is not stored in a KV-cache, but rather in "fast weights" ($W$). As new images come in, the model performs a mini-optimization step (using the Muon optimizer) to update these weights. These weights act as a compressed, implicit 3D representation of the scene.

2. Querying the Memory

To get an explicit 3D model (like Gaussians), the model uses "virtual tokens" to query these fast weights. This decoupling allows the model to scale to massive token counts (millions) while maintaining linear complexity ($O(N)$).

Figure 1: The tttLRM architecture showing how input tokens update fast weights, which are then queried to decode 3D Gaussian Splats.

3. Autoregressive Streaming

Because TTT updates are incremental, tttLRM naturally supports streaming reconstruction. As a camera moves through a scene, the fast weights evolve, and the 3D Gaussian Splat model is refined in real-time.

Performance & Results

The experiments conducted on the DL3DV-10K and Objaverse datasets reveal three major victories:

• Scalability: tttLRM handles 64 views with ease, whereas traditional attention-based LRMs would run out of VRAM.
• Quality: It consistently beats Long-LRM by ~1dB PSNR, providing sharper textures and more accurate geometry.
• Flexibility: The same model can decode into different formats. By changing the query tokens, the authors demonstrated decoding into triplane NeRFs as well.

Figure 2: Qualitative comparison showing tttLRM achieving higher fidelity and fewer artifacts than both optimization-based (3DGS) and feedforward (Long-LRM) baselines.

Distributed Reconstruction Training

To train on such a massive scale, the authors introduced sequence parallelism. Tokens are sharded across multiple GPUs, fast weight gradients are synchronized via DDP, and the final scene is gathered for rendering. This allows the model to utilize 64+ A100 GPUs to digest "millions" of tokens during training.

Critical Insight: The Implicit/Explicit Trade-off

While tttLRM is a massive step forward, the authors honestly note a "slight" quality degradation compared to pure implicit models (like LVSM). This highlights the inherent trade-off in 3D vision: Implicit representations (neural networks) capture detail better but are slow to render, while explicit representations (Gaussians/Meshes) are fast to render but harder to predict from a network.

Conclusion

tttLRM bridges the gap between the speed of feedforward models and the context window of optimization-based ones. By treating model weights as a dynamic 3D memory, it enables the kind of "streaming 3D awareness" required for the next generation of autonomous systems and spatial computing.

Takeaway: The future of 3D vision isn't just bigger Transformers; it's smarter memory management through Test-Time Training.