[arXiv 2026] Matryoshka Gaussian Splatting: Continuous LoD without Quality Sacrifice

Executive Summary

TL;DR: Matryoshka Gaussian Splatting (MGS) solves the fundamental tension between high-fidelity neural rendering and varying computational budgets (mobile vs. desktop). By training Gaussians as a nested set—similar to Russian Matryoshka dolls—MGS enables a single model to be "truncated" at any point to meet real-time FPS targets without the "pop-in" artifacts of discrete LoD or the quality loss of pruning.

Background: Within the 3DGS ecosystem, MGS shifts the paradigm from "post-processing pruning" to "inherently ordered representation." It positions itself as a backbone-agnostic training framework that provides a continuous speed-quality trade-off frontier.

Motivation: The Cost of Scalability

Current 3D Gaussian Splatting (3DGS) models are "all-or-nothing." If you have 5 million Gaussians, you must render 5 million to get a coherent image. Removing primitives post-hoc usually leads to "black holes" or structural collapse because the model hasn't learned to prioritize specific splats.

Existing LoD methods usually rely on:

1. Discrete Levels: Pre-baking 3-5 versions of a scene (wastes memory, causes visual jumps).
2. Hierarchical Structures: Using Octrees (adds complexity to the rendering pipeline).
3. Concurrent Continuous LoD: Often trades off the "peak" quality for the sake of the "low-end" speed.

MGS asks: Can we have our cake and eat it too? Can we reach SOTA peak quality while allowing the model to degrade gracefully to 5% of its size?

Methodology: The Core of MGS

1. The Ordered Representation

MGS ranks every Gaussian by an importance score. While several metrics were tested (Volume, SH energy), descending opacity ($\sigma$) proved most effective. High-opacity Gaussians typically capture the "skeleton" of the scene, while low-opacity ones handle fine-grained residuals.

2. Stochastic Budget Training

The "secret sauce" is the training objective. In every iteration, the model performs two forward passes:

• Full Pass: Renders all $N$ splats to anchor the high-end quality.
• Random Prefix Pass: Renders a subset ${\mathcal{G}}_{\le k}$ where $k$ is sampled randomly.

The loss function balances these two: ${\ell }_{\mathrm{MGS}}:=\ell ({\mathcal{G}}_{\le k};\mathbf{I},\mathbf{c})+\gamma \ell ({\mathcal{G}}_{\le N};\mathbf{I},\mathbf{c})$

This forces the first $k$ splats to be "self-sufficient," capable of representing the scene even without the help of the remaining $N-k$ splats.

Figure 1: MGS enables a continuous spectrum of quality-speed operating points from a single model.

Experiments: SOTA Performance

MGS was tested on major benchmarks like MipNeRF 360 and Tanks & Temples. The results show that even at 10% of the splat budget, MGS remains visually coherent.

Key Results:

• Peak Quality: Matches or exceeds the 3DGS-MCMC baseline (e.g., reaching 28.20 dB PSNR on MipNeRF 360).
• Graceful Degradation: At 5% budget, MGS maintains ~21dB PSNR, while other continuous LoD methods collapse to ~11dB.
• AUC Metrics: MGS dominates in Area Under the Curve for both FPS and splat count, meaning it offers a better trade-off at every possible speed.

Figure 2: MGS (red line) consistently stays above baselines, showing superior quality at all splat counts and frame rates.

Critical Insight: Why Does Opacity Sorting Work?

The authors' ablation study reveals that sorting by opacity is superior to random or SH-energy sorting. This provides a deep insight into 3DGS: opacity is essentially the "visibility weight." By forcing the model to put high-opacity splats at the front of the "Matryoshka" sequence, the training naturally encourages these splats to cover the most critical geometric structures.

Conclusion & Future Work

MGS is a highly practical advancement. Its simplicity—requiring no architectural changes and only one extra forward pass during training—makes it an easy choice for production-grade 3D engines.

Limitations: While the budget is continuous, the current implementation uses a global $k$ prefix. Future iterations might benefit from view-dependent prefixes, where different $k$ values are chosen for background vs. foreground primitives to further optimize the rendering bitstream.

For more details, visit the project page: https://ZhilinGuo.github.io/MGS