Can hierarchical 4D modeling of behavioral decay across sessions detect MCI?

Detecting MCI from how behavior fades over time

Standard screening tools such as MoCA and MMSE require in-person clinical settings and trained administrators, which limits repeated, low-cost monitoring. This study asks whether MCI can instead be detected from ordinary spoken interactions with an Amazon Alexa device, tracked repeatedly over 18 months.

The paper's premise is that cognitive decline shows up as a trajectory rather than a snapshot: facial expression, voice and word choice degrade gradually, and the pattern of that degradation carries diagnostic information. CogDrift-4D is built to represent that trajectory explicitly, rather than averaging it away.

Thirty-five participants, seven sessions, one MoCA cutoff

The dataset contains 35 older adults, each recorded in synchronized audio and video while completing three Alexa-based tasks: answering questions, generating their own queries, and responding to system prompts. The full experiment was repeated seven times per participant at roughly 3-month intervals, giving 105 task-level data points; two participants completed only six sessions.

Labels come from MoCA scores: 18 to 25 is classified as MCI, 26 or above as cognitively healthy. This threshold defines the binary target the model predicts, so every accuracy figure in the paper should be read as performance at that specific cutoff, on this specific group of 35 people.

Why the data are arranged as a 4D tensor

Each sample is stored as a tensor with axes for participant batch, session, time frame within a session, and feature. Keeping session and time as separate axes is the design choice that lets the model treat short-term within-session change and long-term between-session change as different objects.

The baselines show why this matters. A static late-fusion MLP that mean-pools the whole 18-month timeline reached 0.5429 ± 0.09 accuracy, and an isolated per-session Transformer reached 0.5524 ± 0.09 — both near chance. Flattening or isolating the timeline removed the temporal signal the authors consider essential.

Fatigue within a session, drift across sessions

At the intra-session level, the model fits localized least-squares regression slopes to frame-level facial action units and acoustic energy. A downward slope means the participant's expressiveness or vocal energy fades as the task proceeds; the paper calls this acute fatigue and treats the slope itself as a feature.

At the inter-session level, a recursive cross-attention encoder carries a memory of the previous session into the current one, acting as a cumulative fatigue accumulator. A quality gate also scales each channel's representation toward zero when a session looks noisy or uninformative, so the final decision leans on higher-fidelity recordings.

A separate divergence measure compares the visual and acoustic streams: dynamic divergence tracks whether face and voice are out of sync in time, and static divergence tracks whether they are out of sync in emotion. Text enters through sentence embeddings fused early with the physical features.

A 0.733 accuracy ceiling with wide fold variance

In 5-fold cross-validation, CogDrift-4D reached 0.733 ± 0.13 accuracy, 0.716 ± 0.15 F1 and 0.709 ± 0.14 AUC. The strongest external baseline, M-BEHRT, reached 0.6571 ± 0.13 accuracy and 0.6364 ± 0.17 F1; AAMLA and PRA-PoE stayed near 0.57 to 0.58 accuracy.

The authors describe the gap over M-BEHRT as roughly 8 percentage points in accuracy and read it as evidence that token-sequencing Transformers cannot isolate subtle cross-session drift. That is the authors' interpretation; the measured fact is the accuracy difference on this dataset. The ± 0.13 spread also means individual folds vary substantially around the reported mean.

Fatigue features matter most in the ablation

A separate 10-fold cross-validation ablation puts the full model at 0.742 accuracy, 0.731 F1 and 0.724 AUC. Removing the fatigue component dropped F1 to 0.693 and AUC to 0.687; removing divergence gave 0.705 accuracy and 0.689 F1; removing the quality gate gave 0.722 accuracy.

These ablations use a different fold count from the main comparison, so the full model's 0.742 and the headline 0.733 are not the same measurement and should not be compared directly. What the ablation supports is a ranking of contributions within this architecture: fatigue slopes carry the most discriminative weight, divergence and quality gating add smaller gains.

What a 35-person, single-device study can support

The evidence supports a feasibility claim: multimodal behavioral trajectories captured through a consumer voice assistant can separate MoCA-defined MCI from healthy controls better than static or single-session baselines. It does not establish performance in a general clinical population, because the sample is 35 older adults and the label is a single MoCA cutoff rather than a clinical diagnosis.

The paper frames the system as a foundation for home-based, real-time screening. That framing is a stated goal, not a demonstrated deployment result; no home, telehealth or smart-home evaluation is reported in the evidence.

The trajectory, not the snapshot, is the contribution

The most decision-relevant comparison in this paper is between modeling the timeline and discarding it. Static pooling and per-session isolation both landed near chance, while the hierarchical model that keeps within-session slopes and cross-session memory reached 0.733 accuracy. For anyone evaluating behavioral MCI screening, that contrast argues that session structure and within-task fading are the variables worth preserving.

The practical threshold implied by the results is modest: roughly three in four correct classifications at the MoCA 26 cutoff, with fold-to-fold variation wide enough that a single reported run would be misleading. The paper's own ablation ranking — fatigue first, divergence and quality gating second — is the clearest guide to which signals a follow-up system should prioritize.

Sources used in this answer