How can 3D objects be reconstructed from inconsistent multi-view segmentation masks?

Why inconsistent masks corrupt Gaussian optimization

Object-centric 3D Gaussian Splatting needs to know which Gaussians belong to the target object. The usual shortcut is to use 2D segmentation masks, often produced by SAM, to associate Gaussians with objects across views. The paper's core observation is that these masks are inconsistent across views: the same 3D region can be labeled foreground in one view and background in another.

That inconsistency is not isolated 2D noise. Because all views share one 3D Gaussian representation, an erroneous mask propagates through optimization. The paper illustrates this with a controlled case: when only 10% of views (5 of 50) contain mask errors, rendering errors appear in 57% of views (4 of 7). The authors describe this as mask errors destabilizing Gaussian optimization and adaptive density control, producing incorrectly supervised Gaussians that degrade object fidelity.

The paper also reports how common these errors are in practice. Under its automatic mask pipeline, error rates range from 25% to over 80% across Tanks and Temples and Mip-NeRF 360 scenes, for example 30.97% on Barn and 80.51% on bicycle. This is the problem MaRO-GS is designed around: not perfect masks, but imperfect supervision.

Direct object optimization versus full-scene pipelines

Existing object-level 3DGS methods such as ObjectGS, COB-GS, Trace3D, and BEA-GS reconstruct the entire scene before extracting the target object. The paper argues this incurs unnecessary computational overhead when only the target object is needed. Its alternative is to construct an object-masked observation by element-wise multiplying the RGB image with the binary object mask, then optimize Gaussians directly on that masked input.

The paper's Figure 1 comparison on a single LERF-Mask object makes the cost difference concrete: ObjectGS trains for 6,824 seconds and renders in 11 seconds at 43.64 dB PSNR, while MaRO-GS trains for 323 seconds and renders in 5 seconds at 47.64 dB PSNR. Those are single-object illustrative numbers, not the averaged benchmark results, but they show the intended efficiency direction.

The paper notes that simply applying object masks is insufficient, because optimization then depends directly on mask quality and consistency. That dependency is what motivates the three components described next.

Filtering unreliable views before they teach the model

Mask-reliability view filtering identifies and excludes unreliable views from subsequent training. It has two complementary stages. Geometric view consistency measures whether visible Gaussians are consistently supported by the target-object masks across neighboring views, using an object-supported covisibility score. Rather than a fixed absolute threshold, the split threshold is chosen from the largest gap in the sorted score distribution, and accepted only when the low-score cluster is small and clearly separated.

Region-based view validation catches a different failure: views where covisibility looks acceptable but the 2D mask is spatially misaligned due to occlusion or segmentation failure. It computes a per-view hit ratio, the fraction of valid Gaussian projections that fall inside the object mask, and excludes views below a threshold. The final unreliable set is the union of both stages.

The paper is explicit that this is a conservative strategy. On DTU splits, filtering reaches 75% precision on Easy and 100% on Medium and Hard, but recall is low: 24.5% on Easy, 23.1% on Medium, and 4.6% on Hard. In other words, it removes few views but is usually right about the ones it removes, which limits irreversible loss of valid supervision.

Where view filtering helps and where it stops

The benefit of filtering depends on how many views are corrupted. On the DTU Medium split, where corrupted and reliable views coexist, adding both filtering modules raises PSNR from 23.23 to 26.50 dB. On the Easy split, the full filtering configuration improves PSNR from 31.84 to 33.26 dB.

On the Hard split, where most views are corrupted, filtering does not improve PSNR. The paper attributes this to the conservative criterion being unable to remove enough erroneous views without sacrificing supervision, and it shows a corresponding failure case. This is a boundary of the method, not a hidden caveat: the same conservatism that protects valid supervision on Easy and Medium limits gains when errors dominate.

The paper also checks whether filtering harms sparse-view settings. On DTU Scan 12 (Medium) with only six input views, applying both modules does not degrade reconstruction and improves PSNR from 18.82 to 20.71 dB. Sensitivity tests show PSNR varies by only 0.12 dB on LERF-Mask and 0.63 dB on DTU Medium across the tested filtering schedules.

Density control keeps Gaussians on the object

Inconsistent mask supervision also causes unnecessary Gaussian densification around object boundaries and in background regions. Object-supported Gaussian density control suppresses densification outside the target object and removes Gaussians irrelevant to it, keeping the representation compact.

The ablation study shows this is the most consistently useful single module. Mask-guided pruning and densification alone raise PSNR from 41.98 to 42.55 while reducing Gaussian count from 42,846 to 25,805. Combining pruning with appearance-guided local Gaussian merging reduces the count further to 21,157 and achieves the highest FPS, with negligible PSNR degradation.

The paper also reports an ordering effect: the filtering modules need a reasonably localized object representation to work well. Without pruning, geometric view consistency and region-based view validation can incorrectly filter useful views or regions. Region-based view validation alone is especially harmful, dropping PSNR to 37.11. Once pruning refines object support, filtering becomes effective: pruning plus geometric view consistency reaches 42.68 dB, and pruning plus region-based view validation reaches the highest PSNR of 43.01 dB.

Silhouette alignment and the background weight

The silhouette-aligned object loss combines object-focused photometric reconstruction with silhouette-level support alignment. The photometric term weights pixels inside the mask fully and background pixels by a small attenuation factor, while the silhouette term renders a soft silhouette from the current Gaussians and compares it with the target mask.

The background weight matters. When it is zero, the model optimizes only the masked object region and ignores background entirely, which the paper says can reduce training stability because surrounding scene context is not considered. A small background weight improves stability while keeping the object focus. The paper selects 0.01 as the default because it maintains high PSNR and low LPIPS while preserving efficient rendering; at 0.1 the objective becomes dominated by background regions and reconstruction quality degrades.

The full model combines all components and achieves 42.90 dB PSNR with 24,422 Gaussians. The paper notes it does not maximize a single metric, but combines reconstruction quality, compactness, efficiency, and robustness, which is why it is the default configuration.

Benchmark results across three datasets

The main comparison covers LERF-Mask, Mip-NeRF 360, and Tanks and Temples, with all methods evaluated under the same controlled masked-input protocol. The percentages next to each dataset indicate the average object area ratio within the frame: 2.01% for LERF-Mask, 12.54% for Mip-NeRF 360, and 31.35% for Tanks and Temples.

On LERF-Mask, MaRO-GS reaches 42.89 dB PSNR, 0.992 SSIM, 0.008 LPIPS, and 2,081 FPS, compared with 40.67 dB, 0.990, 0.010, and 639 FPS for baseline 3DGS, and 37.65 dB for ObjectGS. The abstract reports the largest PSNR gain as 2.05 dB on this small-object dataset.

On Mip-NeRF 360, MaRO-GS reaches 34.49 dB PSNR and 1,010 FPS, versus 32.99 dB and 250 FPS for baseline 3DGS and 33.46 dB for FDS-GS. On Tanks and Temples, it reaches 30.94 dB PSNR and 397 FPS, versus 30.63 dB and 159 FPS for baseline 3DGS. The pattern is consistent: higher PSNR and much higher FPS than the baseline, with the margin largest where objects occupy the smallest fraction of the frame.

What the Hard split and small scene counts limit

The paper's own limitations section states that severe segmentation failures can still degrade reconstruction quality. The Medium and Hard DTU subsets contain only 3 and 4 scenes respectively, and conservative filtering achieves only 4.6% recall on Hard, so the overall Hard-case improvement cannot be attributed to view filtering alone. Filtering also depends on sufficiently localized Gaussian support during optimization.

These limits matter for how the results should be read. The strongest evidence for view filtering comes from the Medium split, where corrupted and reliable views coexist, and from the Easy split. The Hard split is where the method's conservative design is least able to help, and the small scene counts there mean the Hard-case numbers carry less weight than the LERF-Mask and Mip-NeRF 360 comparisons.

The practical priority that follows from the paper is the ordering of components rather than any single module. Density control must localize the object before view filtering can judge views reliably; filtering alone can over-filter and hurt. The paper's default configuration accepts 42.90 dB rather than the highest single-module PSNR of 43.01 dB because it balances quality, compactness, efficiency, and robustness across datasets.

Sources used in this answer