Doctors vs. UI Experts: The Hidden Risks of Benchmarking Healthcare Tech without Real Users

TL;DR

Is a usability expert a valid substitute for a surgeon when testing medical software? This study demonstrates a significant performance gap between HCI professionals and ENT physicians when interpreting visualized patient data. While HCI pros are great at spotting general UI flaws, their lack of domain knowledge leads to "false positives" and a misunderstanding of what truly matters in a high-stakes clinical environment.

Background Positioning: This research serves as a critical methodological warning within the Healthcare Informatics field, challenging the "discount usability" trend that relies solely on non-expert inspectors for safety-critical systems.

The "Proxy" Problem in Healthcare UX

In the world of software development, recruiting real users for testing is expensive. In healthcare, it’s nearly impossible—doctors are busy and their time is costly. This has led many teams to rely on Usability Inspection Methods (UIM), where HCI professionals simulate being the user.

The authors argue that this "proxy" approach is dangerous for expert systems. If an evaluator doesn't understand a 1.25x multiplier in a medical formula or a TNM cancer classification, they might label a standard medical representation as a "usability issue" while missing actual workflow bottlenecks.

Methodology: The Controlled Walkthrough

The study pitted 13 HCI professionals against 8 Ear, Nose, and Throat (ENT) physicians.

1. The Task: Participants were given 10 screenshots of a tablet-based system visualizing Patient-Reported Outcomes (PRO).
2. The Constraint: Only 5 minutes were allowed for review—mimicking the high-pressure environment of a real consultation prep.
3. The Metric: An 11-item multiple-choice test to see who actually understood the patient's condition.

Note: The system visualizes longitudinal health data for Head and Neck Cancer (HNC) patients to support physician decision-making.

Key Results: Knowledge is Power

The data revealed a stark contrast in performance:

• Physicians (Mean: 9.25/11) vs. HCI Pros (Mean: 7.08/11).
• The TNM Trap: 88% of doctors understood the cancer staging (TNM), while only 31% of UI experts did. To an expert, "pT1 pN2 M0" is a clear status; to a UI evaluator, it looks like cryptic jargon that needs "fixing," potentially leading to unnecessary design changes (False Positives).
• The Blind Spot: Interestingly, doctors performed worse on noticing free-text patient comments. Because they were habituated to systematic data, they overlooked supplemental notes—a "negative" behavior that UI experts correctly identified as a visibility issue.

Figure: The significant difference in scores highlights that professional experience directly impacts the ability to extract meaning from expert interfaces.

Deep Insight: Beyond "Click-Through" Rates

The core takeaway is that Usability is not just about where the button is; it's about semantic density.

• False Positives: HCI professionals might suggest "simplifying" interfaces that are actually efficient for experts, wasting dev resources.
• False Negatives: Because UI experts don't know the clinical "red flags," they might miss the fact that a critical lab value is placed in a way that doctors overlook during a 5-minute rush.

Conclusion & Future Outlook

This paper serves as a vital reminder that while "discount" usability methods are better than nothing, they are not a replacement for domain expertise in safety-critical fields.

The Takeaway for Product Managers: If you are building for surgeons, pilots, or engineers, an HCI degree is not enough to evaluate your product. You must involve the actual "end-user" to avoid a product that is aesthetically pleasing but clinically useless.

Limitations: The study used screenshots rather than the live interactive system and featured a relatively small sample size. Future research should explore "Hybrid Evaluations," where HCI pros and Doctors work in tandem.