MMCORE: Bridging VLM Reasoning with Diffusion Fidelity via Semantic Latents
MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings
MMCORE is a unified multimodal framework designed for high-fidelity image generation and editing. It bridges the gap between Multimodal Large Language Models (MLLMs) and Diffusion Models by using learnable query tokens to extract semantic visual embeddings from a VLM, achieving state-of-the-art results in text-to-image synthesis and interleaved editing.
TL;DR
MMCORE presents a streamlined, resource-efficient framework that connects the high-level reasoning of Multimodal Large Language Models (MLLMs) with the high-fidelity synthesis of Diffusion Transformers. By distilling semantic "visual latents" from a VLM and using them as conditioning signals, it achieves superior text-to-image and interleaved editing performance with significantly lower training costs than training unified models from scratch.
Context: The Divergence of Understanding and Generation
In the current AI landscape, two families rule: Autoregressive (AR) models excel at reasoning/text, while Diffusion models (or Flow-Matching) dominate visual fidelity. Merging them usually results in a "Tax"—either through massive computational overhead or a degradation in the model's ability to "understand" (VQA/OCR) once it learns to "generate."
The authors of MMCORE argue that we don't need to force-fuse these paradigms into a single sequential stream yet. Instead, we can treat the MLLM as a "Semantic Encoder" that provides dense, context-aware visual blueprints for a Diffusion "top-half."
Methodology: The "Scaffold" Distillation
The core innovation lies in how the MLLM is prepared. Rather than relying on a sparse and noisy diffusion loss to teach the MLLM what "visual tokens" should look like, MMCORE introduces Semantic Visual Alignment via Distillation.
- Architecture: They use learnable query tokens that attend to the MLLM's internal states.
- The Scaffold: These tokens are forced to regress toward the feature space of a frozen SigLIP or ViT encoder. This provides a stable, low-variance target.
- Dual-Pathway Conditioning: The diffusion head receives two signals: the MLLM's text embeddings (for lexical detail) and the aligned visual query tokens (for global semantics and multi-image grounding).
Figure 1: The MMCORE pipeline—compressing VLM logic into latent embeddings to steer a diffusion generator.
Handling Interleaved Multi-Image Contexts
One of the hardest tasks in multimodal AI is "interleaved" generation—referencing multiple previous images to create a new one. MMCORE handles this using a block-causal attention mask. It conditions the current generation on the VAE latents of past images but (interestingly) excludes the semantic visual tokens of past frames to prevent optimization instability.
Figure 2: The block-causal mechanism allows the model to "see" history without being overwhelmed by redundant semantic signals.
Experiments and Benchmarks
MMCORE was tested on the DreamBench suite, focusing on counterfactual reasoning (e.g., "a man's eyes level with a woman's mouth") where standard models traditionally fail by falling back on stereotypical priors.
- Efficiency: Achieved SOTA parity using only ~30% of the compute required by native unified architectures like Transfusion.
- Scaling: Performance showed no signs of saturation as batch sizes and sequence lengths were increased, adhering to the "Scaling Laws."
Figure 3: Semantic alignment allows for precise spatial and attribute grounding that mimics human reasoning.
Critical Insight: The Need for an "Omni-Tokenizer"
The authors conclude with a refreshingly honest take on current limitations. Even with MMCORE, there is a "redundancy" between text and visual latents. The visual tokens act as an amendment rather than a full replacement for text.
The path forward, according to the paper, isn't just better alignment, but the creation of an "Omni-Tokenizer": a single representation that supports both the pixel-perfect reconstruction of a VAE and the high-level semantic logic of a ViT. Until then, MMCORE stands as a highly efficient bridge for practitioners looking to combine the best of both LLM and Diffusion worlds.
Summary Takeaway
MMCORE proves that fine-tuning the MLLM backbone with explicit semantic supervision is vastly superior to using frozen adapters. If you want a model that follows complex, multi-image instructions, you must allow the reasoning engine to "learn" how to speak the language of the visual generator.