Can chatbot interviews with older adults screen for perceived cognitive decline?

Interview text as a screening signal

The paper asks a narrow question: can open-ended answers to a chatbot interview reveal whether an older adult reports perceived cognitive decline? Perceived decline here is not a diagnosis. It is a single intake item asked before the interview — "Do you believe you're experiencing cognitive decline?" — with yes or no answers. Participants who said yes were labeled perceived-decline (PD); those who said no were labeled no-perceived-decline (NPD).

That label matters for reading every number that follows. The models are not detecting clinically confirmed mild cognitive impairment or dementia. They are predicting a self-report collected independently of the transcript, and the paper explicitly frames the work as an evaluation of whether interview text contains measurable signals rather than as a clinical detector.

The motivation is scale. Standard neuropsychological assessments are resource-intensive, and the paper notes that tools such as the MMSE and MoCA can be influenced by educational and cultural background and may be insensitive to subtle early changes. A chatbot that conducts a semi-structured conversation could collect language at a distance, which is why the authors test whether that language is worth modeling at all.

Three ways to read a whole interview

The analytic sample contained 153 participants: 100 in the NPD class and 53 in the PD class. The held-out test set contained 31 people, 20 NPD and 11 PD. Every performance figure in the paper comes from that 31-person test set, which is the single most important constraint on how the results should be read.

The authors compared four model families on the interview text: traditional machine learning, frozen transformer embeddings, fine-tuned transformers, and large language model (LLM) reasoning. A frozen embedding means a pretrained language model converts text into vectors without further training on this dataset; a classifier such as a random forest then learns from those vectors. Fine-tuning instead updates the language model itself on the interview data.

LLM reasoning was tested at three levels of granularity. Global Q/A feeds the full aggregated interview to the model. Single Q/A confidence averaging analyzes each question-answer pair separately and combines the outputs. Aggregated Q/A combines intermediate outputs from question-level analysis. This design tests whether the signal lives in the whole conversation, in particular answers, or in a structured combination of both.

Gemini leads text models, within wide intervals

Among text-based methods, Gemini 3.1 Flash Lite with one example per class and Aggregated Q/A produced the highest held-out numbers: 0.742 accuracy, 0.739 balanced accuracy, 0.728 macro-F1, and 0.746 weighted-F1. Balanced accuracy and macro-F1 matter here because the test set is imbalanced, with roughly twice as many NPD as PD participants; plain accuracy alone could look respectable while missing the smaller class.

The paper is careful about this ranking. Eighteen LLM configurations were evaluated on the same held-out test set, with accuracy ranging from 0.484 to 0.742 and balanced accuracy from 0.441 to 0.739. Because the winning configuration was chosen after seeing those test results, the authors call it a descriptive maximum rather than a model-selection result.

The uncertainty analysis reinforces the point. Bootstrap confidence intervals for Gemini Aggregated Q/A overlap substantially with Gemini Global Q/A, GPT-4o, Claude, and the ClinicalBERT plus random forest baseline. With 31 test participants, one additional correct or incorrect prediction moves accuracy by about 0.032. The evidence supports broad differences between model families, not a claim that one LLM configuration is definitively best.

Frozen clinical embeddings outperform fine-tuning

Among non-LLM text methods, frozen transformer embeddings were stronger than fine-tuned transformers. The best non-LLM model was medicalai_ClinicalBERT with a random forest head, reaching 0.645 accuracy, 0.582 balanced accuracy, and 0.583 macro-F1. That suggests pretrained clinical language representations carry useful signal when paired with a non-linear classifier.

Fine-tuned models generalized poorly. Most achieved moderate accuracy, and medicalai_ClinicalBERT fine-tuned reached 0.500 balanced accuracy and 0.392 macro-F1, behavior consistent with near-majority-class prediction. The authors attribute this pattern to the small sample, class imbalance, and the difficulty of end-to-end fine-tuning with limited participant-level data.

This comparison is practically useful for anyone planning similar work: with roughly 150 participants, adapting a large pretrained model end to end did not pay off, while using frozen representations or LLM reasoning did better. The paper's own interpretation is that LLM reasoning and frozen embeddings are more suitable than full fine-tuning in this small-data interview setting.

Structured scores still classify better than text

The numerically strongest classifier in the paper did not use interview text at all. Extra Trees trained on structured questionnaires, cognitive tasks, and personality traits reached 0.871 accuracy, 0.859 balanced accuracy, and 0.859 macro-F1, or 27 of 31 correct predictions. That exceeds the selected ICA-text configuration on the same held-out set.

The authors qualify this comparison rather than treating it as a clean win. Measures such as AD8 and IQCODE assess constructs related to perceived cognitive and functional change, so they are not fully independent of the self-reported outcome. The structured model is therefore best read as a contextual reference baseline, not as an independent predictor that simply outperforms language.

The gap still carries information. Text from an open-ended chatbot conversation was less informative about this self-report than established questionnaire and task scores, which sets a realistic expectation for what interview-based screening would need to add before it could substitute for, rather than complement, structured instruments.

Continuous outcomes show only modest signal

Predicting continuous scores from interview text was substantially harder than the binary classification. Many regression models produced R-squared values near zero or below the mean-prediction baseline, meaning they did not reliably beat simply predicting the average.

The modest positive signals appeared for AD8 (0.218 for neuroticism was the best overall, followed by conscientiousness at 0.209, IQCODE at 0.189, AD8 at 0.183, and FAQ at 0.174). Frozen transformer embeddings with linear regression heads generally gave the strongest and most consistent results across these outcomes.

Letter Comparison performance was poorly predicted. The pattern separates two tasks: coarse classification of a self-reported yes or no is feasible at modest levels, while recovering graded cognitive, functional, and personality scores from the same transcripts is much less reliable.

Direct and autobiographical questions both show signal

An exploratory question-level analysis ranked individual prompts. The top-ranked item was Q41, "Have you noticed any changes in your ability to stay on track in conversations?", classified with an SVM at 0.824 accuracy, 0.700 balanced accuracy, and 0.730 macro-F1 on 17 participants. Q08, an autobiographical prompt about the last job, reached 0.800 accuracy on 10 participants.

Other prompts with strong test metrics included Q04 on career choice and Q33 on getting turned around in familiar areas, each at 0.742 accuracy on 31 participants. The authors note that both direct symptom-related questions and indirect autobiographical or narrative prompts appeared among the stronger items.

These rankings are hypothesis-generating, not confirmatory. The classifier for each question was selected using the same held-out test data, and the questions were then ranked using those same results, a double use of the test set that can inflate the top scores. Effective sample sizes also varied from 10 to 31 across prompts, so raw performance values are not directly comparable without accounting for that denominator.

What clinical translation would require

The paper's own priority is clear from its design: the target is self-reported perceived decline, some interview questions directly query cognitive or functional difficulties and therefore overlap semantically with the outcome, and participants were recruited through Prolific. Proprietary LLMs may also change over time, which limits reproducibility of any specific configuration's numbers.

For readers weighing this approach, the decision-relevant split is between complementing and replacing structured assessment. ICA interview text produced a real but modest classification signal, structured scores were numerically stronger, and continuous outcomes were only weakly predictable. The next step the authors specify is larger, clinically characterized, longitudinal cohorts with validated clinical outcomes, evaluated with questions and classifiers fixed in advance and tested once on an independent cohort.

Until such a cohort exists, the defensible reading is that chatbot interviews capture meaningful signals of perceived cognitive decline and justify continued screening research, while the current evidence does not establish a clinical screening tool. The most promising near-term use is as a scalable, low-burden source of language data alongside validated instruments, not as a stand-alone substitute for them.

Sources used in this answer