UniStitch: Bridging the Divide Between Geometry and Semantics in Image Stitching

TL;DR

UniStitch is a pioneering framework that unifies traditional geometric keypoints with deep semantic features for image stitching. By introducing a Neural Point Transformer and an Adaptive Mixture of Experts, it achieves superior alignment accuracy and unprecedented out-of-distribution (OOD) robustness, effectively solving the "best of both worlds" problem in panoramic vision.

The Great Divergence: Geometry vs. Semantics

For decades, image stitching was synonymous with SIFT and RANSAC. These traditional methods focused on geometric structures—discrete points and lines—that provided high precision in well-textured scenes but crumbled in "texture-less" environments or repetitive patterns.

The deep learning revolution introduced Semantic-based stitching, which utilizes high-level content understanding from CNNs. While these models are robust to lighting and low-texture, they often ignore explicit geometric constraints, leading to "ghosting" artifacts in highly structured areas (like floor tiles or railings) and failing to generalize when tested on datasets outside their training distribution.

Methodology: The Logic of Unification

UniStitch resolves this divergence through a sophisticated architectural bridge.

1. Neural Point Transformer (NPT)

The biggest challenge is that keypoints are sparse 1D lists, while semantic maps are dense 2D grids. The NPT uses a "Transformation-then-Projection" strategy. It leverages PointNeXt to extract high-dimensional point features and then projects them onto structured grids, allowing the geometry to "speak the same language" as the semantic feature maps.

2. Adaptive Mixture of Experts (AMoE) & Modality Robustifier

Not all features are equal in every scene. In a texture-less dark alley, semantic features are king; in a high-contrast skyscraper shot, geometric points are more reliable.

• AMoE: Uses a gated router to weight three experts (Semantic, Geometric, and Heterogeneous).
• Modality Robustifier (MR): A latent-space regularization strategy (random modal dropout) that prevents the model from over-relying on a single modality, ensuring it remains robust even if keypoint detection fails.

Figure 1: The UniStitch pipeline featuring NPT and AMoE modules.

3. FFD-based TPS: Scaling to 4K

Standard Thin-Plate Splines (TPS) are computationally expensive and memory-intensive for high-resolution images. UniStitch introduces Free-Form Deformation (FFD) to decouple deformation complexity from resolution. It predicts flows at a low resolution and uses cubic B-spline interpolation to restore them, enabling 4K stitching on standard GPUs.

Experimental Validation

Quantitative results show that UniStitch isn't just a "hard case" specialist. It consistently beats SOTA methods like RopStitch and StabStitch++ across all difficulty levels (Easy, Moderate, Hard).

Table 1: Performance on UDIS-D dataset showing consistent gains in PSNR and SSIM.

Visually, the difference is clear. Where semantic-only methods produce blurry overlaps on fine structures, UniStitch maintains sharp, continuous lines, leveraging the geometric branch's structural awareness.

Figure 2: Qualitative results comparing UniStitch against traditional and deep learning baselines.

Critical Insight & Future Outlook

The brilliance of UniStitch lies in its recognition that geometric features are not "obsolete" in the age of AI. Instead of replacing them, the authors used points as an inductive bias for the network.

Limitations: The framework currently relies on pre-extracted keypoints (like SIFT or SuperPoint). A truly "end-to-end" version where the keypoint detector is co-optimized with the stitching task could be the next frontier.

Conclusion: UniStitch sets a new standard for image alignment, proving that for tasks requiring high spatial precision, the marriage of classical geometric theory and modern deep learning is more powerful than either in isolation.