[CVPR 2025] VGG-T3: Fitting "Rome" into an MLP for Linear-Time 3D Reconstruction

TL;DR

VGG-T3 solves the "scalability wall" of 3D reconstruction. While current feed-forward models collapse when faced with thousands of images due to $O(n^2)$ attention, VGG-T3 uses Test-Time Training (TTT) to compress the entire scene into the weights of an MLP. It delivers 11x speedups, linear scaling ($O(n)$), and even unlocks "feed-forward visual localization"—all while maintaining high-fidelity geometry.

The Problem: The Quadratic Tax of "Global Memory"

The current era of 3D reconstruction is dominated by feed-forward Transformers (e.g., VGGT, DUSt3R). These models are robust to low overlap and fast motion, but they harbor a fatal flaw: Global Self-Attention.

In these architectures, every image token must attend to every other image token. This creates a "KV space" (Key-Value) that grows linearly in memory but quadratically in compute. If you want to reconstruct a landmark from 1,000 images, you aren't just doing 1,000 times the work of one image—you're doing a million times the interaction work. This leads to Out-of-Memory (OOM) errors and excruciatingly slow inference.

The Insight: MLP as a Fixed-Size Scene Buffer

What if we didn't store the scene as a list of tokens, but as a function?

VGG-T3 draws inspiration from implicit representations (like DeepSDF). Instead of performing softmax attention over an ever-growing KV table, they replace the attention mechanism with a small MLP.

• The Update Phase: At test time, the model takes the image keys ($k$) and values ($v$) and trains the MLP to learn the mapping $kov$.
• The Apply Phase: To retrieve scene information, the query ($q$) is simply passed through the learned MLP weights.

Because the MLP has a fixed dimension, the scene representation doesn't grow with the number of images. The complexity drops from $O(n^2)$ to $O(n)$.

Fig 1. Replacing quadratic global attention with TTT-based linear scaling.

Methodology Core: Engineering the Linearization

Simply swapping Softmax for an MLP doesn't work out of the box. The authors introduced three critical technical refinements:

1. ShortConv2D (Spatial Mixing): Since $K$ and $V$ are linear projections of the same token, the MLP can find a "trivial" solution. VGG-T3 applies a 2D convolution to the $V$ space, forcing the MLP to learn a mapping from a single point ($K$) to its spatial neighborhood ($V^{\prime }$). This injects geometric context.
2. Test-Time Scaling: They discovered that one gradient step is enough for short sequences, but large scenes (1k+ images) require more optimization iterations (2-4 steps) to "cram" the complexity into the MLP.
3. Distributed Inference: Since the TTT loss is a sum of local losses, it is perfectly suited for Data Parallelism. You can shard 2,000 images across 4 GPUs, compute gradients locally, and sync the small MLP weights.

Performance: Fast, Flexible, and "Queryable"

VGG-T3 isn't just a theoretical speedup. In benchmark tests, it outperforms other linear-time methods (like TTT3R) by massive margins and approaches the accuracy of "expensive" quadratic models.

Table: VGG-T3 achieves 33x speedup on 2,000 images compared to the original VGGT implementation.

New Capability: Visual Localization

Because the scene is now stored in an MLP, you can "query" it. If you have a new, unseen tourist photo, you can pass it through the frozen optimized MLP. The model will output the photo's 3D pose relative to the reconstructed scene. VGG-T3 effectively merges "Mapping" and "Localization" into a single feed-forward pass.

Fig 2. Qualitative comparison showing VGG-T3 producing 3D-consistent results while the sequential TTT3R baseline degrades.

Critical Analysis & Outlook

The Takeaway: VGG-T3 is a masterclass in "Post-Training Linearization." It proves we don't need to throw away our massive pre-trained Transformers; we can "refactor" their internal attention mechanisms into linear-time operators.

Limitations: There is still a slight "accuracy gap" compared to full Softmax attention, particularly in wide-baseline settings where the MLP's fixed capacity might struggle to capture every high-frequency detail.

Future Work: The success of VGG-T3 suggests a future where 3D "foundation models" are downloaded as fixed weights, but specialized to a city or a building in seconds via TTT, providing a universal API for both reconstruction and localization.