[CVPR 2025(?)] IOMM: Breaking the Dependency on Paired Data for Unified Multimodal Models

TL;DR

Unified Multimodal Models (UMMs) that both "see" and "draw" usually require massive sets of expensive text-image pairs. IOMM (Image-Only Multimodal Model) flips the script by proving that you can train the "drawing" part using almost exclusively unlabeled images. By combining a Masked Image Modeling objective with a lightweight Residual Query Adapter, IOMM achieves SOTA performance on benchmarks like GenEval (0.89) while being 10x more data-efficient than previous leaders.

Problem & Motivation: The "Paired Data" Bottleneck

Developing a model that understands and generates images (like Gemini or Qwen-Image) is a "grand challenge." Historically, this required millions of high-quality image-text pairs. However, these datasets are:

1. Scarce: Truly high-quality captions are rare.
2. Expensive: Curation costs for proprietary data are astronomical.
3. Inefficient: Current training recipes often fail to teach the model a robust "visual prior," resulting in images that look generic or fail to follow complex instructions.

The authors' insight? Images are their own best teachers. The semantic information needed to generate an image is already hidden within the image itself.

Methodology: The "Sparse-to-Dense" Genius

IOMM employs a two-stage training paradigm.

1. Image-Only Pre-training (The Self-Conditioning Phase)

Instead of using a text prompt, the model uses a reference image as its condition. It extracts features from the image using a frozen MLLM (like InternVL). However, just "copying" an image is too easy. To force the model to learn structure and composition, the authors use Masked Image Modeling (MIM). They randomly mask image patches (up to 85%), forcing the diffusion model to reconstruct a "dense" image from "sparse" clues.

2. The Residual Query Adapter (RQA)

Directly using "understanding" features from an MLLM for generation creates a domain mismatch. The authors introduced the RQA, a tiny 29M parameter module. It uses cross-attention with learned queries to refine the MLLM’s output, acting as a "generative tuner" without risking "catastrophic forgetting" of the MLLM’s original knowledge.

Figure: The IOMM framework showing the self-conditioning loop and the Residual Query Adapter.

Experiments: More with Less

The results are striking. Despite using only 1,050 GPU hours (compared to the massive clusters typically used for SOTA models), IOMM outclasses the competition.

• GenEval Overall: 0.89 (vs. BAGEL-7B's 0.88 and BLIP3-o-8B's 0.84).
• Zero-Shot Editing: Unexpectedly, the model developed superior image editing skills without ever being trained on an "editing" dataset. It maintains original image consistency better than models like UltraEdit.

Table: IOMM vs. SOTA Unified Models. Note the "Ours" section achieving higher scores across Single Obj, Counting, and Position.

The "Mixed-Data" Fine-Tuning Secret

The authors tested six different "data recipes." The winner? Pre-train on 100% images, then fine-tune on a 50/50 mix of images and text-image pairs. This specific recipe ensures the model gains "world knowledge" from the text without losing the "visual fidelity" gained during pre-training.

Critical Analysis & Conclusion

IOMM identifies a critical inefficiency in AI training: our obsession with captions. By shifting the workload to unlabeled image data, the authors have provided a blueprint for more accessible, open-source multimodal research.

Takeaway: If you want to build a better generative model, don't just look for better captions—look for a better way to let the images explain themselves.

Limitations: The authors noted that in larger models (like 2.7B+), image-only fine-tuning can actually hurt prompt-following if not balanced carefully with paired data (the "Mixed-Data" strategy is mandatory).

Visual samples from IOMM-XL, showcasing high-resolution, complex compositional generation.