[CVPR 2025] SO3UFormer: Breaking the Gravity Bias in Panoramic Segmentation
SO3UFormer: Learning Intrinsic Spherical Features for Rotation-Robust Panoramic Segmentation
SO3UFormer is a rotation-robust spherical Transformer designed for panoramic semantic segmentation. By integrating intrinsic geometric priors and removing dependencies on global coordinate frames, it achieves State-of-the-Art (SOTA) performance on the newly proposed Pose35 dataset, effectively preventing performance collapse under arbitrary 3D reorientations.
TL;DR
Panoramic vision models have a "secret" weakness: they assume the world is always upright. SO3UFormer solves this by ditching absolute coordinates in favor of intrinsic spherical geometry. It achieves a massive leap in robustness, retaining 70.67 mIoU under full 3D rotations where previous SOTA methods collapsed to ~25 mIoU.
The "Gravity Shortcut" Problem
Most current panoramic segmentation models are "gravity-aligned." They implicitly learn that the floor is always at the bottom of the Equirectangular Projection (ERP) and the ceiling is at the top. This is a shortcut—the model isn't learning what a floor looks like; it's learning where the latitude is.
When a drone tilts or a handheld camera jitters, this assumption breaks. Standard Transformers, which rely on absolute positional encodings, effectively "look for the floor in the sky," leading to a complete failure of semantic understanding.
Methodology: The Three Geometric Pillars
SO3UFormer moves from extrinsic coordinate learning to intrinsic geometric perception through three core innovations:
1. Removing the Gravity Bias
The first step is simple but radical: remove absolute latitude encoding. By forcing the model to work without a global "Up" vector, it must rely on local patterns and relative geometry.
2. Quadrature-Consistent Attention
Spherical grids (like icosahedrons) don't have uniform node densities. Standard attention treats every node equally, which biases the model toward denser areas. SO3UFormer introduces Area Weights () directly into the attention logit: This ensures the aggregation approximates a continuous integral over the sphere, maintaining consistency regardless of how the grid is rotated.
3. Gauge-Aware Relative Positional Bias
Instead of using a global coordinate system, SO3UFormer defines geometry locally. It projects neighbors onto a tangent plane at each query node and calculates angles relative to local "anchors." By pooling over six discrete in-plane rotations (Gauge Pooling), the model becomes invariant to the local reference frame's orientation.
Experiments & Results: The Pose35 Stress Test
The authors introduced Pose35, a dataset based on Stanford2D3D with random rotations. The results of the SO(3) Stress Test (arbitrary 3D rotations) are eye-opening:
| Method | Base mIoU | SO(3) mIoU (Rotated) |
|---|---|---|
| SphereUFormer | 67.53 | 25.26 (Collapse!) |
| SO3UFormer (Ours) | 72.03 | 70.67 |
Qualitative Impact
In the visualizations, previous models (HealSwin, Elite360) show "blocky" artifacts and misclassify floors as walls when the camera is tilted. SO3UFormer maintains sharp boundaries and correct labels, proving it understands the scene, not just the grid.
Critical Insight & Future Outlook
SO3UFormer proves that mathematical rigor in geometric deep learning isn't just theoretical—it's a requirement for real-world reliability.
Limitations: The model uses index-based resampling during training, which is a discrete approximation. Future work could explore continuous SO(3) representations or "steerable" kernels to achieve even higher fidelity.
Takeaway: If your AI agent (drone, robot, or AR glass) is going to move in 3D space, stop using absolute positional encodings and start respecting the manifold.