[CVPR 2024] Any2Any: Breaking the $O(N^2)$ Barrier in Multi-Modal Remote Sensing Translation

Executive Summary

TL;DR: Any2Any is a unified framework that solves the "missing modality" problem in Earth observation. Unlike previous methods that need a separate model for every pair of sensors (e.g., SAR to RGB, NIR to MS), Any2Any uses a single DiT-based architecture to translate between any arbitrary modalities. By training on the newly released RST-1M (a 1.2M pair dataset), it achieves SOTA results and exhibits "emergent" zero-shot translation for sensor pairs it never saw during training.

Academic Positioning: This work moves remote sensing image-to-image (I2I) translation from technical "silos" into the era of Unified Foundation Models. It effectively reduces the complexity of multi-sensor systems from quadratic $O(N^2)$ to constant $O(1)$.

The Bottleneck: The Quadratic Explosion of Models

In remote sensing, we deal with a "zoo" of sensors: Synthetic Aperture Radar (SAR) for all-weather imaging, Multi-Spectral (MS) for vegetation analysis, and Panchromatic (PAN) for high-res structure.

The status quo has a fatal flaw: Pairwise Training. If you have 5 modalities, you need 20 separate models to cover all directions. This leads to:

1. Massive Storage/Compute: Infeasible as sensor diversity grows.
2. Fragmented Knowledge: A model learning SAR→RGB doesn't benefit from what a model learned in NIR→RGB.
3. Zero Generalization: If you don't have paired SAR-PAN data, you simply cannot perform that translation.

Methodology: Decoupling Representation from Mapping

The authors' core insight is that all modalities are just different "partial observations" of the same underlying geographic scene. To exploit this, Any2Any uses a three-stage pipeline:

1. Unified Latent Manifold

Heterogeneous inputs (different bands, different GSD/resolutions) are projected into a shared latent space $\mathcal{Z}$ via modality-specific VAEs. This standardizes the data geometry before the "heavy lifting" begins.

2. The Shared Semantic Backbone (DiT)

A shared Diffusion Transformer (DiT) acts as the engine. It uses Latent Anchors—paired target latents that "anchor" the denoising process. Instead of predicting noise (which is unstable across sensors), the model uses $x_0$-prediction, directly regressing the clean target latent.

3. Residual Adapters for Calibration

Since VAEs are trained independently, their latent distributions might not perfectly align. The authors introduce Residual Adapters—tiny convolutional branches that correct systematic shifts for specific target modalities. They add $<0.01extM$ parameters but significantly boost fidelity.

Results: More with Less

Quantitative Dominance

Any2Any-L (Large) beats established baselines like BBDM and ControlNet in almost every metric across 14 directions. For instance, in SAR → RGB, it achieves 25.20 PSNR, whereas previous SOTA struggled around 19.50.

The Zero-Shot Miracle

Perhaps the most impressive result is Zero-Shot Generalization. Despite having no paired training data for SAR→PAN or NIR→PAN, Any2Any can generate semantically accurate images by navigating the shared latent space. This proves the model has learned "universal" geographic features.

Critical Insight: Why it Works

The success of Any2Any stems from Transitive Learning. Because RGB is a "pivot" modality connected to SAR, NIR, and PAN in the RST-1M dataset, the shared backbone learns to bridge modalities that were never explicitly paired. The Residual Adapter is the "secret sauce" that ensures the DiT's general semantic predictions are precisely tailored to the target sensor's specific "look."

Conclusion & Limitations

Any2Any is a paradigm-shifting work for Remote Sensing. However, the current resolution is limited by the VAE architecture (standardized to 256x256 or 512x512). Future work could explore Matryoshka-style representations to handle the massive GSD (Ground Sample Distance) variations in satellite imagery more natively.

Final Takeaway: This is a blueprint for building "Universal Translators" for the Earth.