Beyond Text-Dominance: Deciphering the Visual Bias and Internal Mechanics of Omni-modal LLMs
The paper introduces a systematic framework to quantify "Modality Preference" in Omni-modal Large Language Models (OLLMs) using a new conflict-based benchmark and the Modality Selection Rate (MSR) metric. It discovers a paradigm shift from traditional "text-dominance" to a "visual-preference" in modern native OLLMs and achieves SOTA hallucination detection (94% AUROC on POPE) via internal probing.
TL;DR
Is your AI really "listening," or is it just looking at the pictures? Traditional Large Multimodal Models were "text-obsessed," but this study reveals that the new generation of Native Omni-modal Large Language Models (OLLMs) has flipped the script: they are now visually biased. By creating a "tri-modal conflict" benchmark, researchers found that models like Gemini and Qwen-Omni prioritize sight over sound and text, a phenomenon that emerges deep within their neural layers.
The Shift: From Text-Dominance to Visual-Preference
For years, the AI community struggled with "Language Over-reliance," where models would ignore a clear image of a cat if the text prompt hinted at a dog. However, the move toward Native OLLMs—where images, audio, and text are projected into the same latent space—has changed the internal power dynamics.
The authors discovered that when presented with a "tri-modal conflict" (e.g., an image of a bird, audio of a car, and text saying "vaccum cleaner"), models overwhelmingly choose the visual option.
Key Insight: The "Neglected" Audio
Data shows that audio is the "middle child" of the OLLM world. In almost every model tested, the Modality Selection Rate (MSR) for audio remained below 20%, even in models specifically designed to be "omni-modal."
Methodology: Peering into the "Black Box"
How does a model decide which modality to trust? The researchers used Layer-wise Probing. They trained simple linear classifiers (probes) on the hidden states of each layer to see if they could predict the model's final "preference."
The Four-Phase Emergence
The study found that modality preference isn't there from the start. It follows a distinct lifecycle:
- Absent (0-30% depth): Early layers focus on raw feature extraction.
- Emerging (40-70% depth): Preference signals skyrocket.
- Peak (70-90% depth): The "decision" is most clear.
- Declining (90-100% depth): Representations compress for the specific output task.
Experimental Results: Turning Bias into a Diagnostic Tool
The most impressive feat of this research is using these "bias signals" to catch hallucinations. When a model hallucinates, there is a measurable spike in the "interfering modality" probability within its mid-layers.
By monitoring these internal probes, the researchers achieved:
- 94% AUROC on the POPE benchmark for detecting visual hallucinations.
- Significant performance gains across AVHBench (Audio-Video) and AHa-Bench (Audio-Text).
Critical Analysis & Conclusion
This paper provides a sobering look at the "Omni" in OLLM. While we have unified the architecture, we have not yet unified the influence of different senses.
Takeaways:
- Visual Dominance is Real: Most OLLMs trust their "eyes" more than their "ears" or "readings."
- Internal Diagnostics: We don't need external "fact-checkers" for every task; the model's own hidden states often know when they are being misled by a biased preference.
- Future Work: The systematic neglect of audio suggests we need better data balancing or specialized attention mechanisms to ensure OLLMs are truly "balanced" across all modalities.
This work establishes a vital foundation for building Trustworthy AI by understanding not just what a model predicts, but why it chooses one sense over another.