[Stanford Research] Mirage: The Illusion of Visual Understanding in Frontier MMLMs

TL;DR

Current multimodal AI models (GPT-5, Claude 4.5, Gemini 3) are performing a "magic trick": they achieve SOTA results on visual benchmarks by effectively ignoring the images. This paper reveals the Mirage Effect, where models fabricate detailed visual evidence and reasoning traces for images that aren't there. In some cases, a tiny 3B text-only model can beat human radiologists on image-based exams simply by "guessing" based on hidden textual patterns.

The "Mirage" Problem: Seeing What Isn't There

In the current AI landscape, we've focused heavily on hallucinations (making up facts). But the authors identify a more dangerous failure mode: the Mirage.

A Mirage occurs when a model constructs a false epistemic frame. When asked about a missing image, the model doesn't say "I can't see anything." Instead, it describes a non-existent X-ray or scene with absolute confidence.

• The Problem: Our evaluation metrics (Accuracy) and interpretability tools (Chain-of-Thought) can't distinguish between a model that sees an image and a model that imagines one based on the question's text.

Methodology: Quantifying the Illusion

The authors introduced several key metrics and experiments:

1. Mirage-Mode Evaluation: Running models on visual benchmarks without the images.
2. The Super-Guesser: They fine-tuned a 3B Qwen-2.5 model (text-only) on radiology Q&A. Despite having zero visual capability, this "super-guesser" outperformed GPT-5 and human specialists by exploiting structural biases in the dataset.
3. Pathology Bias: In medical contexts, mirages aren't neutral; they are biased toward "worst-case" scenarios like STEMI (heart attacks) or cancer, which could lead to massive over-testing in real clinical deployments.

Why is Benchmark Accuracy Dropping?

The paper presents a sobering fact: most multimodal benchmarks are "tainted." Much of the performance comes from the model's language-prior knowledge rather than its visual "eyes."

• MMMU-Pro & Medical Benchmarks: Models retained 70-80% of their accuracy even when images were removed.
• The "Guessing" Paradox: Interestingly, if you explicitly tell a model to "guess because there is no image," its accuracy drops. This suggests that when a model is in "mirage-mode," it taps into deeper hidden patterns in the prompt that are suppressed when it adopts a conservative "guessing" strategy.

Solution: The B-Clean Framework

To combat this, the authors propose B-Clean. It’s a principled way to "sanitize" benchmarks by:

1. Running multiple models in mirage-mode.
2. Identifying every question that any model can solve without an image.
3. Removing those questions to create a "Vision-Grounded" subset.

When B-Clean was applied, model performance plummeted. In the MicroVQA benchmark, accuracy fell from ~60-70% to just 15-20%. This suggests that 80% of our current "multimodal" progress might just be linguistic pattern matching.

Deep Insights & Future Outlook

• Reasoning Traces are Untrustworthy: You cannot trust a model just because it "explains" what it sees. The explanation can be as fabricated as the answer itself.
• Clinical Risk: In a multi-agent system, if an image fails to upload at step 1, the agents at step 2 and 3 might proceed with a "mirage" diagnosis, causing a silent, catastrophic failure.
• Future Shift: We need Modality-Ablation as a standard unit test. If the Delta ($\Delta$) between "Image-On" and "Image-Off" isn't significant, the model isn't actually using the image.

Conclusion: The AI community must stop treating high benchmark scores as synonymous with understanding. As we move toward 2026 and beyond, the focus must shift from "Bigger Models" to "Better-Grounded Architectures" that refuse to hallucinate a world they cannot see.