CXR-ContraBench: The Hidden Danger of Polarity Reversal in Medical AI
CXR-ContraBench: Benchmarking Negated-Option Attraction in Medical VLMs
This paper introduces CXR-ContraBench, a diagnostic benchmark designed to evaluate "negated-option attraction" in medical vision-language models (MVLMs) for chest X-ray interpretation. The authors demonstrate that models like MedGemma and Qwen2.5-VL systematically prefer negated answer options even when they contradict visual evidence, and propose QCCV-Neg, a deterministic post-hoc verifier that achieves near-perfect accuracy (up to 96.60%) by repairing these polarity reversals without retraining.
TL;DR
Even the most advanced medical Vision-Language Models (MVLMs) suffer from a critical "silent failure": negated-option attraction. This means a model might see a tumor but choose the answer "No tumor" simply because it is drawn to the negative wording. This paper introduces CXR-ContraBench to expose this risk and provides a deterministic fix, QCCV-Neg, which instantly recovers accuracy from ~30% to over 95% without any retraining.
The Problem: When Smart Models Make Fatal Logic Errors
In clinical radiology, a single "not" can be the difference between a correct diagnosis and a life-threatening error. Current MVLMs like MedGemma exhibit a high aggregate accuracy on standard benchmarks, which creates a false sense of security. However, the authors found that when "negated options" (e.g., No Pleural Effusion) are added to the multiple-choice set, models are systematically "attracted" to them, even if the image clearly shows the condition.
Why does this happen? The authors suggest that models may be overfitting to the frequency of negative findings in medical reports or failing to ground the linguistic concept of "negation" in visual evidence—a phenomenon they call presence-side semantic reversal.
Methodology: Auditing Polarity Confusion
The authors built CXR-ContraBench using 135,754 records to stress-test models across three dimensions:
- Internal ReXVQA Slices: Measuring in-distribution stability.
- External Direct Probes (CheXpert/OpenI): Specifically testing if models can identify present findings when a "No X" option is available.
- Ablation of Prompting: Probing whether Chain-of-Thought (CoT) helps (it often doesn't).
The Model Architecture & Failure Point
Figure 1: Comparison between a correct selection and a presence-side semantic reversal where the model is attracted to the negated option despite visual evidence.
Methodology: The QCCV-Neg Verifier
Instead of retraining massive models, the authors propose a "Safety Filter" called QCCV-Neg. It works on a simple but powerful deterministic logic:
- Trigger: If the model selects a negated option (e.g., "No X").
- Verify: Does the question ask for a present finding? Is there a unique positive counterpart (e.g., "X") available?
- Repair: Remap the prediction to the positive finding.
This bypasses the stochastic nature of "reasoning" (CoT), which the authors found can actually amplify contradictions in some cases by making the model focus more on the negated surface forms it sees in the prompt.
Experimental Results: A 65% Accuracy Jump
The results on the CheXpert presence probe were staggering. Models that seemed competent were actually failing over 60% of the time when faced with negation:
Figure 2: Performance atlas showing that QCCV-Neg (Verified) consistently eliminates contradictions and raises accuracy across different model backbones.
| Model | B0 (Baseline) Accuracy | M1 (Verified) Accuracy | Accuracy Gain |
|---|---|---|---|
| MedGemma | 31.49% | 96.60% | +65.11% |
| Qwen2.5-VL | 30.21% | 95.32% | +65.11% |
| GPT-4o | 62.55% | 73.19% | +10.64% |
The study also proved that scaling isn't the solution. Even GPT-4o—one of the most powerful LLMs on earth—showed the highest negation attraction in certain absence-side protocols, demonstrating that this is an architectural or training data bias rather than a lack of "intelligence."
Critical Insight: The Limits of Chain-of-Thought
A common assumption in LLM research is that "Thinking" (CoT) helps models solve complex logic. This paper provides a sobering counter-evidence: for negation in medical images, CoT can be a backfire. By forcing the model to generate more text, it often generates more "negated surface forms," which reinforces its bias to select the wrong answer.
Conclusion & Future Work
CXR-ContraBench is a wake-up call for the medical AI community. It highlights that specialized clinical safety benchmarks are required to uncover "silent" logic failures. The success of QCCV-Neg suggests that for high-stakes medical AI, deterministic rule-based scaffolds might be a necessary layer to protect against the inherent linguistic biases of large generative models.
The next frontier? Understanding the cross-modal representation geometry—essentially, why a model's "pixel-view" of a disease gets disconnected from its "text-view" of a negative statement.