Can paralinguistic cues improve LLM-based Alzheimer's disease detection from speech?

Timing cues are lost before the model sees them

Speech-based Alzheimer's disease (AD) screening is attractive because spontaneous speech can be collected remotely or folded into routine cognitive assessment, and it carries two complementary signals: what a person says, reflecting lexical retrieval and semantic organization, and how they say it, reflecting fluency, timing, and articulation. The paper's starting observation is that the second signal is largely discarded at the transcript stage. Plain transcripts leave pauses and word elongations implicit, so a language model reading them cannot weigh timing against lexical content.

Earlier attempts to fix this have narrow coverage. Mapping timestamp-derived pauses onto punctuation encodes timing, but punctuation also carries syntactic and discourse functions, so the marker conflates timing with linguistic structure. Repetition-based schemes encode only pauses, and the uncapped variant adds more tokens for longer silences without a fixed ceiling. Word-level elongation is usually left as a continuous duration or acoustic feature rather than written into the text. The paper's Figure 1 makes the gap concrete on a single timeline: plain text shows no timing, punctuation and uncapped repetition show pauses only, and the proposed protocol shows both pauses and elongations.

This matters because the paper treats text as the stronger diagnostic channel. In its own ablation, the text-only variant beats the speech-only variant by 9.27 and 12.03 percentage points on ADReSS and by 3.59 and 7.22 points on ADReSSo under cross-validation and leave-one-subject-out respectively. If the stronger channel cannot see timing, paralinguistic evidence has to enter somewhere else, and the paper's answer is to put it into the transcript itself.

Pauses and elongations become bounded text markers

LAPE, short for LLM-Anchored Paralinguistic Enrichment, augments each timestamped transcript word with two kinds of explicit markers. A pause marker is placed before the following word, and its repetition count is set by the gap length: nothing at or below 500 ms, one marker for gaps above 500 ms up to 1000 ms, two for gaps above 1000 ms up to 2000 ms, and three for longer gaps. Elongation markers are placed after the host word and depend on how much the word exceeds a syllable-conditioned reference duration. That reference is a fixed table, from 260 ms for one syllable up to 700 ms plus 150 ms per additional syllable beyond four, initialized from syllable-conditioned duration statistics of ADReSSo ASR word alignments and then rounded and smoothed.

The residual, the observed duration minus the reference, is binned the same way: no marker at or below 200 ms, then one, two, or three markers as the excess grows past 200, 500, and 800 ms. The ceiling of three repetitions is the design choice that separates this protocol from uncapped repetition: longer events are represented, but the marker count cannot grow without bound. Punctuation is retained alongside the markers, so syntactic structure and speech timing stay on separate channels instead of being merged.

The augmented transcript is then passed to a frozen Kimi model together with a fixed task instruction that defines the binary AD-versus-healthy-control classification and explains what the repeated markers mean. The point of the textualization step is not that the model is told about timing in the abstract; it is that timing events sit in the same token sequence as the words, so the language model can model them jointly with lexical content.

Chunking keeps sparse events from being diluted

Textualizing events creates a second problem: the augmented sequence is longer and uneven, and pooling it at the wrong granularity can wash out exactly the sparse events that carry the signal. The paper compares three aggregation levels under the joint protocol. Token keeps subword elements, Word uses lexico-prosodic units, and Chunk mean-pools consecutive word units while keeping each pause, elongation, and boundary unit as a separate singleton.

The results favor the coarsest level. Moving from Token to Word is inconsistent, changing accuracy by 0.00, 2.78, −0.51, and 0.61 percentage points across ADReSS cross-validation, ADReSS leave-one-subject-out, ADReSSo cross-validation, and ADReSSo leave-one-subject-out. Chunk ranks first in all four, exceeding the better fine-grained alternative by 2.73, 2.78, 0.68, and 2.41 points, an average of 2.15. Relative to Token, its mean gain grows from 1.71 points under cross-validation to 4.29 under leave-one-subject-out.

The authors read this pattern as evidence that pooling consecutive word units suppresses speaker-specific redundancy while preserving sparse paralinguistic events, and that complementary event markers must remain salient rather than being diluted across a long token sequence. That reading is an interpretation of the ablation pattern, but the pattern itself is consistent: the benefit of chunking is larger under the stricter subject-wise evaluation, where speaker-specific shortcuts are less available.

Joint markers beat either event type alone

The protocol comparison isolates which cues matter. Against a basic transcript, adding pause markers alone improves ADReSS by 3.63 points under cross-validation and 1.86 under leave-one-subject-out, but changes ADReSSo by 0.00 and −1.20 points. Adding elongation markers alone changes the four settings by −1.18, −3.70, −0.02, and −1.20 points. Neither single-event protocol is reliable on its own, and the elongation-only variant is worse than the basic transcript in every setting.

The joint protocol, which encodes both event types, improves over the basic transcript by 5.45, 5.56, 1.82, and 3.62 points, averaging 4.11, and exceeds the better single-event protocol by 2.74 points on average. Its gain is larger under leave-one-subject-out than under cross-validation on both datasets, most visibly on ADReSSo, where the gain is 3.62 points versus 1.82. The authors explain this as complementary roles: pauses mark interruptions, while elongations capture prolonged speech in context, which makes the joint encoding more robust to held-out speakers.

This is the paper's most decision-relevant comparison for anyone considering a transcript protocol. The value is not in adding more timing information but in adding both event types, and the advantage of doing so is largest in the evaluation that holds out entire speakers.

Speech features are scaled against a text anchor

The fusion stage explains why the model does not simply concatenate everything. LAPE keeps four blocks: the LLM text representation, a locally aggregated Whisper speech representation aligned to the same word and event units, and two utterance-level blocks, a handcrafted eGeMAPS vector covering prosody, phonation, and voice quality, and a mean-pooled WavLM representation. Text enters as the direct anchor and is never rescaled. The three speech-derived blocks pass through NormGate, which standardizes each block using statistics from the training data of each run, rescales by the square root of its dimensionality so higher-dimensional blocks do not dominate, and predicts a per-recording scalar gate from all four blocks.

The gate is bounded between 0.5 and 1.5 and starts at 1 because the final predictor layer is zero-initialized. Each auxiliary block therefore retains 50 to 150% of its standardized baseline magnitude, and training learns a sample-dependent modulation rather than a fixed weighting. The ablation shows the local Whisper block never reduces accuracy when added to text, with gains of 0.91, 2.78, 0.60, and 0.00 points, averaging 1.07. The utterance-level blocks without NormGate help only ADReSSo leave-one-subject-out and are neutral or worse elsewhere; adding NormGate raises accuracy by 0.91, 2.78, 0.62, and 1.21 points over that variant, averaging 1.38, with a larger mean gain under leave-one-subject-out than cross-validation, 2.00 versus 0.77 points.

The asymmetry is the point. Local acoustic evidence tied to specific words and events is consistently useful, while recording-wide voice characteristics need adaptive scaling relative to text before they help. The authors present NormGate as addressing unequal modality contributions, and the ablation supports that role specifically for the utterance-level blocks.

The lead holds under subject-wise evaluation

LAPE ranks first across all four primary settings, reaching 98.18% and 94.01% accuracy under participant-level cross-validation and 95.37% and 91.57% under leave-one-subject-out on ADReSS and ADReSSo. Its lead over the strongest listed baseline spans 2.21 to 8.37 points, averaging 5.55. The paper focuses on accuracy because macro F1 differs from it by at most 0.01%, and it argues that the small official test sets produce coarse metric increments and split sensitivity, whereas participant-level cross-validation and leave-one-subject-out evaluate every participant out of sample.

The stricter evaluation does cost accuracy: leave-one-subject-out reduces LAPE's accuracy by 2.81 points on ADReSS and 2.44 on ADReSSo, yet its margins remain 7.41 and 4.22 points, compared with 8.37 and 2.21 under cross-validation. Accuracy on ADReSSo is 3.80 to 4.17 points lower than on ADReSS, consistent with the fully automatic speech-only setting, although the authors note that dataset differences preclude attributing this gap to transcription alone.

That caveat matters for reading the comparison. ADReSS supplies official manual transcripts while ADReSSo provides speech only, so the two benchmarks differ in more than one respect, and the paper does not claim the transcription condition alone explains the gap.

The protocol choice matters more than the fusion operator

The paper's evidence points to a specific priority for speech-based AD detection with LLMs: the transcript protocol and the aggregation granularity carry more of the gain than the fusion mechanism. Joint event markers add 4.11 points on average over a basic transcript, chunk aggregation adds 2.15 points on average over the better fine-grained alternative, and NormGate adds 1.38 points on average over utterance-level blocks without it. The largest single lever is encoding both pause and elongation events rather than one.

The scope conditions are equally specific. All results come from ADReSS and ADReSSo under participant-level cross-validation and leave-one-subject-out, with frozen Kimi and Whisper backbones and a fixed syllable-conditioned duration table derived from ADReSSo alignments. The elongation reference is a coarse task-specific baseline, and the authors describe extending LAPE toward richer paralinguistic cues and distillation into smaller LLMs as future work, so the reported numbers describe this configuration rather than a deployed screening tool.

For a reader deciding what to carry forward, the practical split is between event coverage and event preservation. Adding only pauses, or only elongations, is not reliably better than a plain transcript in these results; adding both and keeping each event as its own chunk is where the accuracy advantage appears, and it appears most clearly when entire speakers are held out.

Sources used in this answer