Global attention dominates runtime as views grow
Feed-forward 3D foundation models such as VGGT, HunyuanWorld-Mirror, and Pi3 predict camera poses and dense scene geometry from unposed images in a single forward pass, removing per-scene optimization. They still alternate frame-wise attention with global attention so each image can draw on observations from other views, and that global attention scales quadratically with the number of input views. In the authors' measurement, global attention accounts for 88% of VGGT's end-to-end runtime at 512 views.
Two cheaper operators are available. Sparse attention keeps token-level softmax interactions but only over selected regions, while linear attention aggregates global context at linear cost. Applying either one uniformly ignores the fact that different pretrained stages may respond differently to approximation, which is the question the paper sets out to answer.
Sensitive stages also concentrate their attention
The authors profile every global-attention stage in the three pretrained backbones using two independent diagnostics: sensitivity to linear conversion, and how concentrated the attention mass is. Across all three models a recurring depth-wise pattern appears. Early stages spread attention across many tokens and tolerate linear replacement surprisingly well. Middle-to-late stages are more sensitive to linear replacement, and their attention is highly concentrated on fewer tokens.
That alignment is the paper's central observation: the stages that most need faithful softmax interactions are also the stages where sparse selection can cover most of the relevant attention mass. The authors frame this as turning an apparent trade-off into an allocation principle rather than as a proven causal mechanism.
Linear-centered stages plus tile-sparse stages
Zebra3D ranks global stages by their single-stage linear-conversion error, assigns a requested fraction to linear-centered attention, and gives the remaining stages tile-sparse attention. In the xL/yS shorthand, x is the fraction of layers using linear attention and y is the sparsity of attention rows in the remaining layers; the selector retains a key-tile fraction of 1 minus y. Reported operating points include 0.5L/0.6S, 0.5L/0.8S, 0.7L/0.6S, and 0.7L/0.8S.
In sparse stages each image is partitioned into 8x8-patch tiles of 64 tokens. Selected tiles keep full softmax interactions, while omitted tiles are summarized as mean key/value landmarks so unselected context is not simply discarded. Special-token keys stay visible, special-token query rows keep dense attention, and every patch query retains its own self key. Conversion preserves compatible pretrained parameters, including the input encoder, QKV and output projections, normalization layers, positional encoding, and prediction heads.
A closed-form fit avoids retraining from scratch
For each linear stage, the frozen dense model is run on 13 held-out multi-view calibration scenes. Teacher softmax outputs and converted linear outputs are accumulated across scenes, tokens, heads, and views, and a centered affine correction is fitted by a regularized least-squares solve in float64. This requires one teacher forward pass and one solve per converted stage and does not update inherited weights by gradient descent.
An optional post-conversion training stage adds task supervision plus output and relational feature distillation. The complete objective, RelKD, combines task supervision with output, feature-L1, token-cosine, and cross-view relational losses. Ablations on RealEstate10K camera pose show the difference: at VGGT 0.7L/0.8S, ground-truth supervision alone gives 0.777 AUC-30, while RelKD reaches 0.805.
Near-dense accuracy at 79 to 94 percent sparsity
Across all three backbones, Zebra3D reports strong reconstruction and camera-pose accuracy at high attention sparsity, with training-free variants remaining close to their dense counterparts. After post-conversion training, Zebra3D-VGGT reaches 80.5% RealEstate10K AUC@30 at 94% mean attention sparsity, within 1.1 percentage points of dense VGGT fine-tuned on the same data and schedule.
The paper's summary table also places the converted models against other efficient approaches on reconstruction and pose metrics. These numbers compare different operating points and should be read with their sparsity settings attached, since a more aggressive linear fraction buys speed at a cost that grows on harder data.
Speedup scales with the number of input views
End-to-end speedups increase with input view count, reaching up to 7.1x at 1,024 views with packed Triton kernels. At the 0.7L/0.8S setting the paper reports respective end-to-end speedups of 6.2x, 5.5x, and 7.1x at 1,024 views across the three backbones. The kernels fuse Q/K normalization, positional rotation, and channel-softmax feature mapping for linear stages, and pack the key/value sequence into tile-major order for sparse stages.
On ScanNet-50 at 1,000 frames, the 0.7L/0.6S configuration reaches the lowest Chamfer distance of 0.447 at 3.54x speedup, while 0.7L/0.8S reaches 5.60x speedup with a Chamfer distance of 0.460. These are quality-speed operating points, not a single ranking.
Wide baselines punish the 0.7L setting
The main benchmarks are dominated by video-like capture, where consecutive views share a narrow baseline and attention is most likely to concentrate on a few neighboring frames. The appendix tests Tanks and Temples, ETH3D, and DTU, averaging over view counts from 8 to 64. There, every 0.5L configuration stays within 3% of its dense backbone in normal consistency, with the largest gap 2.9% for HYWM 0.5L/0.8S on Tanks and Temples.
The 0.7L configurations are less forgiving. Pi3 degrades most: 0.7L/0.8S loses 0.076 NC1 on Tanks and Temples, a 10.4% relative drop, and 0.054 on ETH3D, 6.4%. The authors connect this to their placement analysis, since Pi3 has the latest sensitive band, so raising the linear fraction encroaches on it soonest. They therefore treat 0.5L as the operating point that transfers to wide baselines and 0.7L as a speed-first setting validated mainly on video-like benchmarks.
Choose the linear fraction by baseline width
The paper's most useful distinction is not a single speedup number but the split between its two operating regimes. If the target workload resembles densely sampled video trajectories, the 0.7L settings deliver the largest gains, up to 7.1x at 1,024 views, with accuracy that the authors validate mainly on those benchmarks. If views are sparse or widely separated, the evidence points to 0.5L, where normal consistency stays within 3% of dense backbones across the wide-baseline tests.
The conversion itself is cheap by design: a closed-form affine fit on 13 calibration scenes, no gradient updates, and preserved pretrained parameters. That makes the layer assignment, not retraining, the main decision variable. The authors list input-adaptive allocation and broader hardware support as future work, so the fixed per-backbone layer tables remain the practical starting point.
Sources used in this answer
Zebra3D: Efficient Hybrid Foundation Models For Scalable 3D Reconstruction
Converting pretrained feed-forward 3D models into layerwise sparse-linear attention hybrids reaches up to 7.1x end-to-end speedup at 1,024 views, with Zebra3D-VGGT at 80.5% RealEstate10K AUC@30 at 94% mean attention sparsity, within 1.1 points of dense VGGT.