[CVPR 2025/ArXiv] MedSPOT: Why Your MLLM Fails in the Radiology Suite
MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI
MedSPOT is a novel benchmark designed to evaluate Multimodal Large Language Models (MLLMs) on sequential visual grounding within clinical Graphical User Interfaces (GUIs). It features 216 task-driven medical workflows (597 annotated keyframes) across 10 imaging platforms, establishing a new SOTA evaluation standard using a strict "early-termination" protocol.
TL;DR
The industry is racing toward AI agents that can "navigate" software like humans. However, MedSPOT reveals a sobering reality: even our best Multimodal Large Language Models (MLLMs) are effectively "blind" when dropped into the dense, high-stakes environment of clinical software. By introducing a strict sequential evaluation protocol across 10 medical platforms, this benchmark shows that current SOTA models collapse when a task requires more than a single click.
Background: The Gap in GUI Grounding
While LLMs can pass the Bar Exam, their multimodal siblings struggle to find a "Save" icon in a DICOM viewer. Existing benchmarks like ScreenSpot or Mind2Web treat GUI interaction as a series of independent coin flips. In reality, medical workflows are causally linked chains: if you fail to click the "Open File" icon in Step 1, you can't possibly select the patient record in Step 2.
The "Sequential Fragility" Phenomenon
The core contribution of MedSPOT is its Early-Termination Protocol. Unlike previous metrics that give partial credit for correct steps, MedSPOT terminates a task the moment a model misses a target.
The results are startling:
- Generalist Collapse: Models like Llama 3.2-11B and Qwen2-VL achieved 0% Task Completion Accuracy (TCA).
- The SOTA Gap: The strongest model, GUI-Actor, dropped from a 65% first-step accuracy to just 43.5% TCA for full workflows.
The decay from Step-1 Accuracy (S1A) to Task Completion Accuracy (TCA) illustrates how errors compound in multi-step clinical tasks.
Methodology: Workflow-Aware Grounding
MedSPOT treats the GUI as a "state-dependent" environment. The authors curated 216 tasks involving 10 professional tools (e.g., RadiAnt, 3D Slicer, ITK-SNAP).
The Failure Taxonomy
Why do they fail? The authors provide a brilliant systematic diagnosis of "Model Pathology":
- Toolbar Confusion: Mistaking global navigation for local task elements.
- Edge Bias: A technical "glitch" where models over-predict coordinates near the screen borders.
- Small Target Error: The inability to "see" tiny, 24x24 pixel icons—a common feature in medical software.
MedSPOT spans diverse clinical categories, from DICOM viewers to segmentation tools, ensuring models cannot simply overfit to one UI style.
Detailed Results & Insights
The paper highlights a fundamental "Representational Mismatch." Current MLLMs use patch-based tokenization (e.g., 14x14 patches). If a medical icon is smaller than a single patch, the model loses the signal in the noise.
Comparative analysis showing the massive performance delta between GUI-specialized models (Aguvis, GUI-Actor) and general-purpose LLMs.
Critical Analysis & Conclusion
Takeaway
MedSPOT proves that parameter scale is not a panacea for spatial grounding. A 70B model with poor spatial resolution is less useful in a GUI than a 7B model specifically trained for pixel-level precision.
Limitations
- Monolingual: Currently English-only.
- Click-Centric: Does not yet evaluate "Drag-and-Drop" or "Scroll," which are vital for navigating 3D MRI volumes.
- Risk-Unaware: Clicking the "Delete Patient" button is penalized the same as clicking the "Help" icon.
The Future
For AI to assist radiologists, we need models that understand Hierarchy and Context. MedSPOT sets the stage for the next generation of "Native GUI Agents" that don't just see the screen, but understand the workflow.
Source: MedSPOT: A Workflow-Aware Sequential Grounding Benchmark for Clinical GUI. Project Page