Can an LLM agent outperform ophthalmology residents at pre-consultation history-taking?

A head-to-head test of who takes the history

The study asks whether a large language model (LLM) can do the information-gathering part of an eye clinic visit — the pre-consultation history — as well as the residents who normally do it. Between May 10 and June 10, 2025, 175 patients with non-emergency appointments at a tertiary ophthalmic hospital in Guangzhou were screened; 3 declined and 172 were randomized 1:1, 86 to the LLM agent and 86 to ophthalmology residents with at least three years of experience. All randomized patients completed the trial, and no adverse events were reported.

The primary outcome was the MedHistory composite score, a 0–100 rubric built for this study from ophthalmologist focus groups and adapted from the Objective Structured Clinical Examination format. It weights three domains: history-taking (92 points) for completeness and clinical reasoning in the dialogue and chart, communication (4 points) for empathy and clarity, and documentation (4 points) for terminology accuracy and dialogue–chart consistency. A score of 70 or above was defined as high quality. Two masked evaluators with more than five years of ophthalmology experience scored the encounters, and their agreement was reported as excellent.

Because the score is dominated by the history-taking subscale, the headline difference is mostly a statement about thoroughness of questioning and reasoning, not about bedside manner or chart formatting. That distinction matters when reading every result below.

Logic trees, not a general chatbot

The agent was not a general-purpose chatbot prompted to act like a doctor. It used a dual-agent architecture built on a locally deployed Qwen-2.5-72B model that had been fine-tuned on ophthalmology records. A main agent runs the multi-turn conversation, guided by "skills" — executable inquiry units that encode branching clinical logic, organized into symptom-oriented logic trees. For a red eye, for example, the tree directs questioning toward visual change, pain, discharge and ocular trauma.

An auxiliary agent works alongside it, extracting and summarizing key clinical features as feedback and drafting a preliminary report with a patient summary, test recommendations and primary diagnoses. A documentation module then standardizes terminology into a semi-structured chart. Memory modules supply short-term dialogue context and long-term domain knowledge drawn from the authors' Comprehensive Ophthalmic Dataset, and an emotion-recognition mechanism adjusts tone and pacing.

The authors describe the design goal as separating comprehensive data collection from premature diagnostic closure, and they state explicitly that the system was built to prioritize high-fidelity history acquisition and documentation support rather than autonomous diagnosis. Before the trial, a proof-of-concept benchmark had the agent and four comparator models each process 20 real outpatient cases; the agent scored highest on history-taking quality against all four comparators.

Higher scores came with longer conversations

On the primary outcome, the agent's mean MedHistory score was 80.73 versus 56.94 for residents, an unadjusted difference of 23.79 points. After adjustment for age, sex and pre-consultation duration, the difference was 17.67 points (95% CI, 14.07 to 21.27; P < 0.001). The advantage came from history-taking (adjusted difference 17.41 points) and, to a small degree, communication (0.36 points; P = 0.03). Documentation did not differ significantly between groups.

The item-level breakdown explains where the points came from: residents captured the chief complaint well, while the agent more consistently elicited structured details such as precipitating factors and clinical course that residents less often recorded.

The cost is time. Median pre-consultation duration was 11.2 minutes (IQR 8.7 to 13.5) for the agent versus 3.1 minutes (IQR 2.4 to 4.4) for residents, P < 0.001. The authors note that the agent's systematic, exhaustive strategy contrasts with residents' experience-driven heuristics for targeted inquiry, and that the longer interaction may itself contribute to better history quality. In a pragmatic trial these factors co-occurred inside the intervention, so the trial cannot separate how much of the gain came from questioning strategy versus extra time.

Patients liked the agent, except for reassurance

Overall satisfaction was comparable between groups, as were conversational naturalness and having concerns addressed before follow-up questions. The differences appear on specific items rather than in the aggregate. The agent was rated higher for patience (median 5 vs 4) and perceived empathy (5 vs 3), both P < 0.001, and patients assigned to it were more likely to consider AI-led pre-consultation a viable alternative (5 vs 3) and to trust that their privacy would be protected (5 vs 4; P = 0.005).

Residents were rated better at reducing patient anxiety (4.5 vs 3; P = 0.005). The authors read this as a reminder that building deep therapeutic trust remains a distinctly human competency, and they note that privacy was not a prominent concern among participants, contrary to some prior reports.

These are 5-point Likert ratings from an 8-item questionnaire developed with patient focus groups, so they describe perceived experience, not clinical outcomes. The split pattern is the useful part: the agent's advantage is in patience and perceived empathy, while the resident's advantage is in reassurance.

The gap narrows once the eye exam enters

The exploratory diagnostic analyses are where the comparison changes shape. Working from history alone, the agent had higher primary diagnostic accuracy, with an estimated F1 difference of 0.18 (95% CI, 0.08 to 0.30; P = 0.001); the abstract reports F1 of 0.85 versus 0.68. When textual descriptions of ocular signs — slit-lamp findings, visual acuity and intraocular pressure — were added, residents gained substantially more than the agent (estimated F1 difference 0.24; 95% CI, 0.15 to 0.32; P < 0.001). The incremental benefit attributable to physical examination was lower for the agent by an estimated 0.19 F1 points (95% CI, −0.31 to −0.07; P = 0.003).

The authors call this a "stage effect" and suggest residents relied on examination findings to compensate for briefer initial histories, a pattern they say may reflect real-world workflow pressures favoring rapid assessment of observable findings. They also report that higher MedHistory scores were positively associated with better diagnostic performance at the history-only stage.

Two cautions belong here rather than at the end. These diagnostic analyses are exploratory, and the reference standard was the initial workup rather than final diagnostic confirmation, so the authors themselves caution against reading clinical superiority into them. The F1 comparison also uses a different task and denominator from the MedHistory score and should not be substituted for it.

Test recommendations converged; cost projections diverged

Test recommendation performance did not differ significantly between groups before or after multiplicity correction. At the signs-integration stage, the quality of treatment plan formulation was also comparable (median 3 vs 3; P = 0.75).

The finer-grained comparison favored the agent. Its projected test-related expenses and wait times were closer to specialist-defined standards, with a Hodges–Lehmann difference between residents and the agent of ¥30 (95% CI, ¥21.0 to ¥87.0; P < 0.001) for expenses and 0.0 minutes (95% CI, 0.0 to 1.1; P < 0.001) for wait times. The authors describe residents as using a heuristic, satisficing approach that was generally effective, while the agent's systematic method showed advantages at this finer granularity.

Read together with the diagnostic results, this suggests the agent's contribution is consistency in what gets asked and ordered, not superior clinical judgment at the point of synthesis. The expense and wait-time figures are projections derived from institutional data, not observed spending.

One center, one language, one specialty

The trial ran at a single tertiary ophthalmic hospital, in Chinese, with non-emergency appointments, and the MedHistory rubric was newly designed in-house for this study. The authors state that further adaptation may be required for other populations and health systems. The locally deployed open-source model was chosen for data privacy but may have capped performance, particularly on medical terminology; the authors report that in a post hoc review of transcripts from 50 randomly selected participants, speech-to-text artifacts fell from 3.7 to 0.5 per 100 characters after documentation optimization.

The trial was open-label, since the intervention's nature made blinding patients and clinical assistants impossible, though evaluators and the statistician were masked to allocation and hypotheses. The authors flag that long-term monitoring is needed to assess novelty bias. Interactions were also limited to text and voice, and the intervention's effect on physician cognitive load and workflow efficiency in high-volume settings was not assessed.

Ophthalmology is a favorable test case because its symptom vocabulary is well defined, which the authors say minimizes the semantic ambiguity common in broader medical practice. That makes the result a benchmark for the approach rather than evidence that it transfers unchanged to less structured specialties.

Standardize the history, keep the exam human

The paper's own conclusion is narrower than "the agent beat the residents." It supports a hybrid workflow: the agent standardizes clinical data acquisition before the physician encounter, and clinicians direct their expertise toward physical examination, complex diagnostic synthesis and shared therapeutic decisions. The evidence for that split is the pattern across outcomes — a large history-quality advantage, no documentation or test-recommendation advantage, a diagnostic edge from history alone that shrinks once ocular signs are available, and a reassurance advantage that stayed with residents.

The practical threshold the data suggest is about where the agent's marginal value is largest: structured, detail-heavy history capture, where it gained roughly 17 adjusted points and elicited precipitating factors and course details residents often missed. Where the task shifts toward interpreting examination findings or calming an anxious patient, the resident comparison is even or favors the resident.

The unresolved trade-off is time: 11.2 versus 3.1 minutes per patient. Whether that extra time is affordable, and whether it produces better downstream diagnoses rather than better-scored histories, is what the authors say future multicenter work on clinical efficiency and patient outcomes should determine.

Sources used in this answer