MOIR: Rebalancing Vision-Language Models via Information-Level Routing

TL;DR

Researchers from Georgia Tech have introduced MOIR (Multi-modal Information Router), a plug-and-play module that fixes "modality dominance"—the tendency of AI to ignore images and rely solely on text. Unlike previous methods that just tweak attention weights, MOIR identifies "weak" data channels and refills them with information from other modalities, making models more robust and visually grounded.

Background: The "Lazy" VLM Problem

Modern Vision-Language Models (VLMs) are impressive, but they often "cheat." Studies show that many VLMs can answer questions about an image even if the image is replaced with random noise. This modality dominance (usually favoring text) happens because text is semantically dense while visual tokens can be sparse or ambiguous.

The authors argue that simply telling a model to "pay more attention to the image" doesn't work if the visual features themselves lack the necessary information density. You can't squeeze blood from a stone, and you can't extract reasoning from uninformative tokens.

The Core Insight: Information vs. Attention

The fundamental innovation of MOIR is shifting the intervention from the Attention Mechanism to the Information Level.

1. Identifying the "Vague" Tokens

MOIR uses Singular Value Decomposition (SVD) to analyze token embeddings. It looks for channels that contribute little to the principal components of the representation. If a token's representation is dominated by weak or noisy channels, it is flagged as "less informative."

2. Adaptive Information Routing

Instead of discarding these weak tokens, MOIR acts as a "router." It takes high-density information from the complementary modality (e.g., text) and injects it into those weak visual channels via learnable gates ($\alpha$). This creates a balanced, information-dense representation before the LLM ever sees the data.

Fig 1: Comparison between standard VLM fusion (a) and MOIR (b). MOIR routes info to ensure balanced attention flow.

Methodology: Precision Grafting of Data

The process follows a clean mathematical logic:

1. Decompose: $ilde{X}_A=U_A{\Sigma }_A(V_A{)}^{o}p$ to find the importance score $S_d^A$ for each channel.
2. Select: Identify the bottom $k^{\prime }$ channels that are effectively "dead weight."
3. Route: Mix in features from the other modality: ${\bar{\mathbf{F}}}^A=\alpha \cdot {\mathbf{F}}^B+(1-\alpha )\cdot {\mathbf{F}}^A$

This ensures that by the time the tokens reach the LLM decoder, the "information disparity" has been neutralized.

Fig 2: Detailed routing mechanism identifying less informative (lighter) tokens and filling them ($T_1,T_4$).

Experimental Proof: Real-World Robustness

The impact of MOIR is most visible when things go wrong.

• Visual Corruption Test: When images were replaced with noise, standard models still gave the same answer 62.13% of the time (proving they weren't "looking" at the image). MOIR slashed this to 29.63%, forcing the model to actually use the visual input.
• Performance Gains: On the VizWiz benchmark (VQA for the blind), MOIR improved accuracy from 43.50% to 47.00%.
• Information Rank: The "Rank $\Delta$"—a measure of how much information is packed into the embeddings—consistently rose for both modalities.

Table 1: MOIR consistently reduces MDI (Modality Dominance Index) and increases Rank across multiple backbones.

Critical Perspective: Why This Matters

The value of MOIR lies in its Inductive Bias. It acknowledges that LLMs are "lazy" learners that will always take the path of least resistance (usually text priors). By modifying the "availability" of information at the source, MOIR makes it impossible for the model to ignore one modality in favor of another.

Limitations & Future Work

While MOIR is effective, the $k^{\prime }$ (routing ratio) requires tuning. The study found $0.10$ to be the "Goldilocks" zone—too much routing might blur the unique semantic boundaries of each modality, while too little fails to fix the dominance issue. Future iterations could benefit from a dynamically predicted $k$ based on input noise levels.

Conclusion (Takeaway)

MOIR proves that modality dominance is an information problem, not just an attention problem. By engineering the information density of tokens before they enter the LLM, we can build VLMs that are not just accurate, but truly multi-modal.