Why better AI did not simply mean better teams
Medical AI is often validated as if algorithmic sensitivity and clinician oversight add up: a better model plus a careful reader should produce better care. This study tested that additive assumption in neonatal pneumoperitoneum, free air in the peritoneal cavity that signals a time-critical surgical emergency, often after necrotizing enterocolitis or intestinal perforation.
The authors argue that the relevant question is not standalone accuracy but what happens to the human–AI team when the algorithm is wrong. They define systemic resilience as the capacity to keep diagnostic integrity under algorithmic failure, measured through clinicians' behavior on AI-wrong cases: accepting the error or correctly overriding it.
A crossover stress test of two AI reliability levels
Fourteen readers — six pediatric radiologists, three neonatologists and five radiology residents — interpreted 125 external-validation radiographs across three arms: unaided (41 cases), reliable AI (40 cases) and error-injected AI (44 cases). The design was a multi-reader, multi-center crossover with a six-week washout between sessions, and readers were not told that two different AI reliability levels existed.
The reliable model reached an AUC of 0.861 on its allocated subset, with sensitivity 0.77 and specificity 0.96. The error-injected model was engineered to be unreliable, with balanced accuracy 0.44 (sensitivity 0.40, specificity 0.47), built from 18 false positives, 6 false negatives and 20 correct predictions. The analysis covered 1750 interpretation events using generalized linear mixed models with crossed random effects for reader and case.
Trainees followed wrong AI advice most often
On cases where the reliable AI was wrong, radiology residents accepted the incorrect suggestion in 52.0% (13/25) of reader-case events where they could follow or override. Pediatric radiologists did so in 20.0% (6/30), a difference that remained significant after Bonferroni correction (P = 0.016); the comparison with neonatologists at 33.3% (5/15) did not reach significance (P = 0.326).
This error acceptance is the paper's clearest measure of automation bias: the reader adopts an incorrect automated cue. It is a different quantity from overall accuracy. Residents' overall accuracy under reliable AI was 0.89, not a statistically significant improvement over their unaided baseline (OR 0.92, 95% CI 0.32–2.70; P = 0.885), which the authors attribute to error acceptance counteracting correct AI predictions.
Experts overrode flawed AI rather than absorbing it
When the AI was error-injected, neonatologists correctly overrode it in 91.7% (66/72) of instances, the pattern the authors call sentinel behavior. Pediatric radiologists overrode 85.4% (123/144) and residents 81.7% (98/120).
The statistical support is an interaction rather than a simple group difference: neonatologists' response to error-injected AI diverged from the pediatric radiologist reference pattern (interaction OR 4.16, 95% CI 1.26–13.77; P = 0.020). Post-hoc contrasts against each group's own unaided baseline did not survive multiple-comparison adjustment, and the neonatologist group contained only three readers, so the authors label the finding hypothesis-generating and call for prospective replication.
Longer reading time marks verification effort
Disagreement with AI advice was associated with longer interpretation time across all groups (β = 0.892, SE = 0.174, t(1043) = 5.12, P < 0.001). Group-stratified marginal slopes were 1.77× longer for pediatric radiologists (95% CI 1.44–2.18), 1.95× for neonatologists (95% CI 1.46–2.59) and 1.49× for residents (95% CI 1.18–1.88) on discordant versus concordant cases.
The between-group difference in this slowdown was not significant (F(2, 1096) = 1.32, P = 0.267), so the authors treat the extra time as a similar workflow cost across expertise levels rather than as proof of a distinct expert mechanism. Their interpretation is that contradicting AI prompts deliberation, but the reading-time measure is a proxy, not a direct observation of reasoning.
Years of experience did not predict accuracy
After controlling for specialty, clinical experience was not a significant predictor of accuracy (P = 0.771), and neither gestational age (P = 0.542) nor birth weight (P = 0.969) influenced correctness. The authors read this as evidence that the behavioral differences came from domain-specific training rather than cumulative seniority.
That distinction matters for how the results should be used. The relevant split in this study is between trainees and specialists in the roles they occupied, not a smooth gradient of years served. Pediatric radiologists stayed stable across conditions, with no significant gain from reliable AI (P = 0.358) and no significant loss from error-injected AI (P = 0.490).
Where the resilience claim stops
The specialist cohort is small: three neonatologists, with 375 independent decision points in that subgroup. Leave-one-neonatologist-out refits kept the interaction directionally consistent (odds ratios 2.04 to 2.64) and a reader-clustered GEE confirmed it (P = 0.018), but the authors still describe the result as an initial characterization that needs larger multi-center replication.
The saliency-map analysis is explicitly exploratory and descriptive. Because opening the heatmap was user-initiated, usage was subject to selection effects, so the reported 100% error-override rate when neonatologists viewed the map cannot support a causal claim about explainability. The study also used binary AI outputs without confidence scores and imposed no time limit, so it does not test how probability displays or time pressure would change these behaviors.
Resilience as the team-level metric
The paper's practical priority is a shift in what gets measured. A high standalone AUC was not sufficient evidence of clinical safety here, because the same reliable model produced the highest error acceptance among trainees. The authors propose evaluating human–AI teams on adversarial resilience — error-acceptance and correct-override rates on AI-wrong cases — alongside accuracy.
The scenario split that follows from the evidence is between a reliable model and a deliberately failing one. Reliable AI offered limited gains to experts and exposed trainee over-reliance; error-injected AI revealed expert override capacity. The authors also release an open-source educational sandbox intended to inoculate clinicians against automated errors, aimed at the deskilling and never-skilling risks raised by these behavior patterns.
Sources used in this answer
Expertise modulates automation bias and sentinel behavior in human–AI collaborative diagnosis of neonatal pneumoperitoneum
High AI reliability induced automation bias in trainees, who accepted 52.0% of incorrect suggestions, while three neonatologists correctly overrode 91.7% of errors from a deliberately flawed model.