CROSSMATH: Exposing the "Modality Gap" in Vision-Language Reasoning
Do Vision-Language Models Truly Perform Vision Reasoning? A Rigorous Study of the Modality Gap
This paper introduces CROSSMATH, a rigorous benchmark designed to evaluate whether Vision-Language Models (VLMs) perform genuine visual reasoning or rely on textual shortcuts. Using 2D intersecting math puzzles in semantically equivalent text, image, and hybrid formats, it reveals a massive performance gap where vision-only inputs cause state-of-the-art models (like Qwen3.5-Plus) to collapse.
TL;DR
Do current Vision-Language Models (VLMs) actually "see" the logic in an image, or are they just literate text models with a visual attachment? This paper introduces CROSSMATH, a benchmark of 2D intersecting math puzzles. The verdict is sobering: models like Qwen3.5-Plus that dominate text-based math collapse when the exact same problem is presented as an image. The authors demonstrate that this isn't a perception problem—it's a reasoning failure that can be mitigated through specialized RL-based post-training.
The "Illusion" of Multimodal Intelligence
Modern VLMs follow a standard recipe: a Vision Encoder (like CLIP) plus a massive LLM backbone. While they perform well on general benchmarks, researchers suspected a "Textual Shortcut." If a question contains enough textual context, the model ignores the image.
To test this, the authors created CROSSMATH. The task is a 2D grid of intersecting equations (think crossword puzzles, but with math). To solve it, a model must:
- Understand 2D Spatial Layout: Which numbers belong to which equation?
- Iterative Deduction: Solve Equation A to find a variable for Equation B.
The genius of CROSSMATH is Information Equivalence. Every puzzle exists in three formats: a Markdown table (Text), a Grid Image (Vision), and a combined version. Because the information is identical, any performance drop in the "Image-only" mode directly measures the Modality Gap.
Figure 1: CROSSMATH uses semantically equivalent formats to isolate the impact of the visual modality.
The Modality Gap: A Precipitous Drop
The results are a "wake-up call" for the industry. Even elite models like Qwen3.5-Plus (92.8% accuracy in text) fail miserably in vision (12.4% accuracy).
Statistical analysis revealed three critical insights:
- It's not OCR: Models could transcribe the images into Markdown with high accuracy. The failure happens after perception—at the reasoning stage.
- Vision as a Distractor: Surprisingly, adding an image to a text prompt often lowered performance. The visual noise seemingly interfered with the LLM's clean textual logic.
- The Depth Wall: As the "hops" (number of logical steps) increased, vision-based accuracy plummeted much faster than text-based accuracy.
Table 2: Comparison across modalities shows the extreme performance decay when models rely on visual input.
Bridging the Gap: Post-Training with GRPO
The authors didn't just diagnose the problem; they proposed a cure. Using a pipeline of Supervised Fine-Tuning (SFT) followed by Group Relative Policy Optimization (GRPO), they trained a Qwen3.5-9B model specifically on image-only reasoning paths.
Key to their success was a Position-Weighted Reward. Instead of giving a simple "Correct/Incorrect" score, they gave higher rewards for solving "deep" variables that required multiple reasoning hops.
The Result? The post-trained 9B model jumped from 3.2% to 50.4% macro accuracy in image-only mode. More importantly, this wasn't just "overfitting" to the puzzle style. The model showed improved performance on external benchmarks like MMMU and MathVerse, proving that teaching the model how to ground symbols in space benefits general multimodal intelligence.
Critical Analysis & Conclusion
This paper proves that scaling the LLM is not enough. A 397B parameter model (Qwen3.5-397B) still struggled with visual reasoning that a much smaller 9B model mastered after targeted post-training.
Takeaways for the Industry:
- Architectural Bottleneck: Current vision-to-language projectors (linear layers or MLP) might be too "shallow" to preserve the structural constraints needed for multi-step logic.
- The Need for "Vision-CoT": Future models need training data that explicitly links visual coordinates to reasoning steps, rather than just image-caption pairs.
The "Modality Gap" reminds us that while our models can read like scholars, they still perceive the world like infants. Benchmarks like CROSSMATH provide the necessary "eye exams" to move toward true, unified multimodal reasoning.