How do AI explanations and self-assessment shape novice reliance calibration?

Reliance calibration measures alignment, not trust

Reliance calibration is the paper's organizing construct: the degree to which a user's acceptance or rejection of AI suggestions matches whether those suggestions are actually correct over an interaction session. A well-calibrated user accepts suggestions when the AI is right and rejects or edits them when it is wrong. A miscalibrated user systematically over-relies or under-relies.

The study separates this from raw acceptance. Across the two AI conditions, participants accepted suggestions 75.0% of the time, modified them 20.2%, and rejected them 4.8%. Acceptance was lower with explanations (72.0%) than with predictions alone (79.0%). Those rates describe behavior; calibration asks whether that behavior followed AI correctness.

The authors summarize miscalibration with an RC score, where lower is better calibrated, and split it into over-reliance and under-reliance. In this sample the two were very unequal: mean over-reliance was 0.31 (SD 0.24) against mean under-reliance of 0.04 (SD 0.12), and a paired one-sided Wilcoxon test confirmed participants over-relied significantly more than they under-relied (p << .001).

Without feedback, users must self-monitor

Most prior calibration work assumes explicit performance feedback: task-level signals telling users whether their judgment or the AI's suggestion was correct after each decision. Feedback and explanations then both act as learning signals that let users update their mental model of AI reliability.

The paper targets settings where that signal is absent. A clinician using a diagnostic tool or an analyst labeling medical text may finish a whole session without learning whether accepting or overriding the AI was appropriate. In such environments users must draw on internal metacognitive resources, reflecting on what they know and how confident they are, to decide whether to accept or override.

The authors argue this mechanism is fragile for novices, whose domain expertise is insufficient to independently verify AI outputs or interpret explanations. That is the gap the study tests: whether explanations can substitute for missing feedback when users cannot check the AI themselves.

A clinical extraction task with limited feedback

The study ran a between-subjects experiment with 110 participants recruited through Prolific, a research-oriented crowdsourcing platform. Participants extracted clinical words and phrases in six categories, including Signs & Symptoms, Lab Value, Medication, and Disease Disorder, from anonymized patient discharge summaries. For each document they had to understand the clinical category and then correctly identify and highlight the matching text spans.

Four conditions varied AI support: no AI, no AI plus a medical dictionary, AI suggestions alone, and AI suggestions with explanations. Participants produced 26,460 decision points in the form of annotated text. Task accuracy was higher in the AI-assisted conditions (AI: M = 0.429, SD = 0.257; AI + Exp: M = 0.454, SD = 0.260) than in the non-AI conditions (No AI: M = 0.211, SD = 0.226; No AI + MD: M = 0.180, SD = 0.212).

The authors note that the task was chosen because it needs specialized knowledge, remains achievable with effort, and is complex enough to warrant AI recommendations. The limitation they attach to this design is that the clinical categories were selected to be distinguishable; if categories were hard to tell apart even with instructions and explanations, misunderstanding could bias results.

Explanations left calibration essentially unchanged

The central test of explanations used a Bayesian linear mixed-effects model with participant random intercepts and explanation condition and clinical category as fixed effects. The explanation coefficient was negligible and non-credible (beta = -0.02, 95% CI [-0.05, 0.02]), with the posterior centered near zero and residual variance unchanged. Explanation condition also did not attenuate the difficulty differences across clinical categories.

Active engagement did not rescue this. Within the AI + Exp condition, neither dwell time on the confidence score (beta = 0.00, 95% CI [-0.01, 0.02]) nor dwell time on the medical dictionary (beta = -0.01, 95% CI [-0.03, 0.01]) was associated with lower RC scores.

One narrower signal went the other way. The per-item reliance indicator was positively associated with AI correctness in both conditions, weakly in AI only (rho = 0.20, p < .001) and slightly stronger with explanations (rho = 0.27, p < .001); a Fisher r-to-z test confirmed the correlations differed significantly (z = -2.52, p = .012). The authors read this as explanations modestly improving sensitivity to AI correctness, even though the aggregate calibration score did not move.

Over-reliance drifts upward across the session

The session-trajectory analysis regressed RC scores on task sequence with random intercepts and slopes for participants. The overall RC score tended to increase over the session, but that effect was not significant. The components moved in a more telling way: over-reliance increased as the session progressed (estimate = 0.04, 95% CI [-0.01, 0.09]; P(beta > 0) = 0.94), while under-reliance decreased significantly (estimate = -0.01, 95% CI [-0.03, 0.00]; P(beta < 0) = 0.97).

The authors describe this pattern as a drift toward uncritical acceptance of AI suggestions over time. Random-slope variance exceeded the average slope, meaning individual trajectories varied substantially rather than all users following one path.

Miscalibration also depended on what was being annotated. Adding clinical category as a fixed effect showed Lab Value tasks produced the highest miscalibration (estimated mean about 0.60, beta = 0.33, 95% CI [0.28, 0.38]), while Diagnostic Procedure showed moderate miscalibration (beta = 0.07, 95% CI [0.02, 0.13]). Category reduced residual variance from 0.25 to 0.21; adding document ID yielded no evidence and left these coefficients unchanged.

Task understanding predicts selectivity better than confidence

Perceived competence was uniformly high before the session (M = 4.33, SD = 0.61) and declined slightly by the end (M = 4.03, SD = 0.86), with the largest drop in the AI-only condition (M = -0.47, SD = 0.88). Using pre-session perceived competence as the moderator, a directional test gave moderate support for the hypothesis that higher perceived competence predicts lower RC scores (posterior probability 0.88, evidence ratio 7.08).

The interaction result ran against the authors' expectation. The explanation by perceived-competence interaction was positive but non-credible (beta = 0.05, 95% CI [-0.02, 0.13]), with posterior mass mostly positive, which the authors interpret as explanations possibly narrowing rather than amplifying the calibration advantage of high-competence participants.

The sharper split came from separating confidence from understanding. In a Bayesian logistic mixed model over 12,798 items, confidence did not predict appropriate reliance in the acceptance channels, and more confident participants were less likely to reject or edit wrong suggestions (beta = -0.33, 95% CI [-0.73, 0.05]; P(beta < 0) = 0.95). Understanding behaved oppositely: higher understanding increased both rejection and editing of incorrect suggestions (beta = 0.51, 95% CI [0.09, 0.94]; P(beta > 0) = 0.99) and addition of missed entities (beta = 0.47, 95% CI [0.05, 0.89]; P(beta > 0) = 0.99), with no credible effect on accepting correct suggestions.

Monitor category-specific reliance, not global trust

The paper's most actionable distinction is that miscalibration is not a single user trait. The same user working with the same AI system behaved differently depending on the clinical category, and Lab Value tasks stood out as the most miscalibrated. The authors attribute this to numeric values being visually salient and easy to identify, leading users to over-rely without checking whether a recommendation meets the clinical definition of a lab value in context. One participant in the AI condition described skimming easy categories such as medication and lab value without taking the AI's highlighted text into account.

That reading supports a design priority rather than a general call for more transparency. Because explanations presented uniformly with every suggestion placed the evaluation burden on users who lacked the epistemic resources to use them, the authors propose metacognitive monitoring functions that track behavioral signals such as acceptance rates, editing frequency, and dwell time on explanations, and flag shifts within a knowledge domain. Such functions do not require ground truth during the task, unlike performance feedback.

The scope conditions matter for applying this. Participants were Prolific crowdworkers, not a wide demographic, and the study did not account for cognitive load or implicit motivation. Within those bounds, the practical threshold the paper offers is behavioral: a spike in acceptance rate within a knowledge domain, or a collapse in post-recommendation editing relative to a user's own baseline, is the signal worth acting on, especially where explanations alone did not correct the drift.

Sources used in this answer