[CVPR 2025] ITO: Cracking the Modality Gap with Training-Time Fusion

Executive Summary

TL;DR: The "modality gap" is a recognized phenomenon where images and texts, despite being "aligned" in CLIP, remain segregated in the embedding space. ITO (Image-Text as One) solves this by using a lightweight fusion module during training that is discarded at inference. This approach forces the vision and text encoders to learn a truly unified semantic space, resulting in SOTA performance across 26 datasets and billion-scale benchmarks without any extra computational cost at runtime.

Background Positioning: This work is a significant "architectural-training" hybrid innovation. It moves beyond simple contrastive scaling (like SigLIP or OpenCLIP) to address the geometric structure of the latent space, positioning itself as a robust upgrade for the next generation of foundation model backbones.

The "Modality Gap" Problem

Even when a CLIP model achieves high retrieval accuracy, UMAP visualizations often show two distinct "islands"—one for images, one for text. This suggests the model is using modality-specific "shortcuts" rather than understanding a shared concept.

Previous attempts to bridge this either:

1. Added permanent fusion layers: These are slow and break the "dual-encoder" efficiency (e.g., FIBER).
2. Shared parameters: This often limits the capacity of individual encoders (e.g., AlignCLIP).

ITO's core insight is that fusion should be a teacher, not a permanent component.

Methodology: Synergy of Alignment and Fusion

ITO's architecture consists of two main pillars:

1. Multimodal Multiple Alignment

Instead of a simple one-to-one image-text pair, ITO generates multiple augmented views of both images and texts. This creates a "dense" supervision web (one-to-many and many-to-many), pushing the model to find invariant semantic features across different perturbations.

2. Training-Time Multimodal Fusion

This is the "secret sauce." During training, a 2-layer Transformer takes the tokens from both encoders and fuses them. A contrastive loss is applied to these fused tokens. To minimize the loss, the original encoders must output representations that are highly compatible for fusion.

The Magic of $\lambda$: The fusion loss ($L_{Fusion}$) acts as a structural regularizer. By adjusting its weight $\lambda$, the authors balance raw discriminative power (from alignment) with geometric integrity (from fusion).

Experimental Results & Scalability

ITO was tested on everything from 3M samples to the 1-Billion sample DataComp dataset.

• Zero-Shot Supremacy: On Laion100M, ITO beats CLIP by 2.6% on average across 26 classification tasks.
• MLLM Backbone: When used as the vision backbone for LLaVA-1.5, ITO-trained encoders significantly improved reasoning performance on VQAv2 and MMVet, proving that a "merged" embedding space makes it easier for LLMs to interpret visual data.
• Efficiency: Training ITO takes about 1.4x the time of CLIP, but Zero change to inference speed.

Deep Insight: Fusion as a Stabilizer

One of the most striking findings in the paper is ITO's training stability. Standard contrastive learning (CLIP/SLIP) often suffers from "early saturation"—performances peaks and then starts to drop (overfitting) if trained too long.

The authors found that fusion prevents this. By forcing a unified structure, the model is less likely to overfit to modality-specific noise. The UMAP visualizations below show the dramatic difference:

(Left: CLIP's separated space; Right: ITO's integrated space)

Conclusion & Key Takeaways

ITO proves that we don't need heavier models for better alignment; we need better training objectives.

• Takeaway 1: Instance-level matching is not enough for true cross-modal understanding.
• Takeaway 2: Training-time fusion is a "free lunch" that improves representation quality and training stability without affecting deployment speed.
• Limit: While ITO works great on billion-scale web data, the benefits of text-augmentation (sampling sub-descriptions) diminish at scale, suggesting that data diversity eventually replaces the need for complex augmentation.

Future Outlook: ITO provides a blueprint for training efficient dual-encoders that possess the "understanding" of much heavier fusion-based models. Expect to see this "training-only fusion" logic applied to Video-Audio and Multi-sensor pretraining soon.