[arXiv 2026] OmniJigsaw: Solving the Bi-Modal Shortcut in Multi-Modal Reasoning
OmniJigsaw: Enhancing Omni-Modal Reasoning via Modality-Orchestrated Reordering
OmniJigsaw is a self-supervised reinforcement learning (RL) post-training framework that enhances omni-modal reasoning by using a temporal reordering (jigsaw) proxy task. Built on the Qwen3-Omni-30B-A3B-Instruct baseline, it achieves state-of-the-art results across 15 benchmarks, including a +4.38 gain on MLVU-Test and +2.50 on MMAR.
Executive Summary
TL;DR: OmniJigsaw is a breakthrough self-supervised framework that uses temporal "jigsaw puzzles"—reordering shuffled video and audio clips—to train omni-modal models via Reinforcement Learning (RL). By introducing a novel Clip-level Modality Masking (CMM) strategy, it forces models to bridge the gap between sight and sound, leading to double-digit improvements in complex reasoning benchmarks without requiring manual annotations.
Background Location: This work represents a major step in the RL Post-Training lineage (following DeepSeek-R1 and Tulu 3), but moves beyond text into the Omni-Modal domain. It is a "SOTA-enhancing" methodology that proves lightweight proxy tasks can compete with resource-heavy supervised learning.
The Core Conflict: The "Bi-Modal Shortcut"
Most multi-modal models are lazy. When presented with a video of someone explaining a recipe, the model might solve a reordering task by just listening to the speech (Audio) or just watching the ingredient changes (Video).
The authors identify this as the "Bi-modal Shortcut Phenomenon". If the model can solve the puzzle using only one stream, it never learns the deep, "collaborative" reasoning needed for truly intelligent agents. This "path of least resistance" results in weak representation learning for the non-dominant modality.
Methodology: Orchestrating the Chaos
To break the shortcut, OmniJigsaw doesn't just shuffle clips; it orchestrates them. The paper proposes three strategies, with the third being the "Secret Sauce":
- Joint Modality Integration (JMI): Both audio and video are provided. (Susceptible to shortcuts).
- Sample-level Modality Selection (SMS): The model chooses the most informative modality for the whole video.
- Clip-level Modality Masking (CMM): The model acts as a "modality selector" for each individual clip. It might keep the video for Clip A, the audio for Clip B, and both for Clip C.
Why CMM Works
By masking modalities at a granular level, CMM creates an information bottleneck. To put the puzzle back together, the model is forced to perform "cross-modal semantic stitching." It must remember a visual cue from a masked-audio clip to link it to a sound cue in a masked-video clip.
Scalability via Data Filtering
A jigsaw puzzle is only useful if it's solvable. Randomly cut videos often lack clear temporal flow (e.g., a static "talking head").
OmniJigsaw uses a two-stage filtering pipeline:
- Stage 1 (Heuristic): Simple algorithms check for motion (Mean Absolute Difference) and audio quality (Spectral Flux).
- Stage 2 (Semantic): A smaller MLLM (Qwen2.5-VL-7B) acts as a judge, using Chain-of-Thought (CoT) to verify if the video has a "directionally unambiguous chronological progression."
Experimental Results: Proving the Synergy
The researchers applied OmniJigsaw to the Qwen3-Omni-30B baseline using GRPO (Group Relative Policy Optimization).
- Video Mastery: +4.38 gain on MLVU-Test.
- Audio Depth: +2.50 gain on MMAR.
- Omni-Modal Collaboration: +1.70 gain on OmniVideoBench.
A key ablation (Fig 5) shows that their Accuracy-dependent Discount Factor (rewarding perfect reordering 5x more than partial success) acts as a catalyst, preventing the model from settling for "good enough" solutions.
Critical Insight: More is Not Always Better
The paper concludes with a warning: integration of modalities without a proper "bottleneck" (like CMM) can actually degrade performance compared to uni-modal training. The "Bi-modal shortcut" is a real threat to foundation model training. Only by intentionally making the task "hard" through masking does the model develop the "muscles" for complex omni-modal integration.
Limitations & Future Work
While powerful, the current proxy task is limited to fixed-length clips. Future iterations could explore variable durations or spatial jigsaws (shuffling parts of the frame) to further push the boundaries of spatial-temporal reasoning.
Senior Editor's Note: OmniJigsaw proves that we don't need more human-labeled data; we need smarter ways to let models play with the data we already have.