Does language model information help imagined speech decoding more than listened speech?

The question and the short answer

The paper asks how much a language model helps when the neural evidence comes from imagined speech rather than listened speech. It uses a dataset in which the same participants either listened to or imagined the same two poems, so the two conditions can be compared under one pipeline.

The answer is asymmetric. Both conditions improve when neural scores are combined with language-model next-word probabilities, but imagined speech peaks at a larger language-model weight and shows a larger gain over neural-only decoding. The authors interpret this as language priors becoming more valuable as neural evidence becomes less reliable.

Why imagined speech is the harder input

Imagined speech is harder to decode because neural responses during imagination are weaker and less reliably time-locked than responses evoked by an external speech signal. Imagined-speech datasets are also relatively small, partly because imagination has no directly observable output and its timing is hard to measure precisely.

The paper's own numbers show this gap. With the proposed decoder, listened MEG reached 14.17% top-1 and 46.96% top-5 word accuracy, while mapped imagined MEG reached 7.05% top-1 and 31.14% top-5. The imagined accuracy is about half the listened accuracy at top-1, so the two conditions supply neural evidence of different strength.

Two places where language enters the pipeline

Language information is added at two stages. During training, a contrastive neural decoder aligns MEG representations with two frozen targets: frame-level wav2vec 2.0 hidden states for acoustic information and word-level GPT-2 hidden states for contextual language information. The encoder is a 1×1 spatial convolution followed by a causal dilated temporal convolutional network producing representations at about 50 Hz.

During inference, a neural-constrained beam search fuses the decoder's word scores with GPT-2 next-word probabilities. At each word position, MEG selects the top-k candidate words, GPT-2 scores only those candidates given each beam's own decoded history, and the two normalized scores are combined with a weight α. Setting α to 0 ranks candidates by MEG alone; setting it to 1 keeps MEG's candidate set but ranks within it entirely by the language model.

The decoder improves both conditions

The contrastive decoder substantially improved cross-subject word decoding over the previous decoder for listened MEG, and the improvement transferred to mapped imagined responses. It exceeded its controls for all six imagined-to-listened mappings at both top-1 and top-5, with all p < 0.001.

The controls matter for reading the result. With shuffled neural–target pairs, listened accuracy fell to 4.12% top-1 and imagined to 3.93%, and on all-zero input accuracy fell to chance. The authors note that the acoustic objective appears to prevent a shortcut: with the language objective alone, the decoder scored above chance even on zero input by defaulting to frequent-word targets, whereas frame-level acoustic alignment encouraged it to track the neural input.

Teacher forcing bounds the benefit of context

To measure how much linguistic context could contribute in principle, the authors first let GPT-2 condition on the true preceding words and score the full vocabulary. In both conditions, accuracy rose with α until α was close to 1, and the two conditions converged at high α.

That analysis bounds the benefit but does not represent realistic decoding. With the correct history, the language model dominates ranking and neural evidence adds little. It also assumes a history that is unavailable at inference, which is why the paper turns to beam search where the model conditions on its own decoded hypotheses.

Imagined speech shifts the optimal balance

Under neural-constrained beam search at beam width 3 and candidate-set size 10, both conditions showed an intermediate optimum in α, and joint scoring improved over MEG-only ranking in both (p < 0.001). Both also stayed above a random-candidate null across α, so neural candidate selection mattered in each condition.

The endpoints differ. For listened MEG, ranking candidates by MEG scores alone (α = 0) outperformed ranking them by the language model alone (α = 1). For mapped imagined MEG, that order reversed. Across candidate-set sizes, the optimal fusion weight α* was higher for mapped imagined than for listened MEG (p < 0.01), and the relative gain over MEG-only ranking at the selected α was larger for imagined speech (paired test across subjects, p < 0.001).

The candidate-set size explains part of the mechanism. With small k, MEG constrains decoding by limiting the language model to a few neurally selected words, so neural information remains present even at α = 1. As k grows, that constraint weakens and the language model can choose among more candidates, so removing the relative MEG scores becomes more costly. MEG therefore contributes both by selecting candidates and by ranking words within that set.

What the comparison does not cover

The evaluation is limited to two poems, a closed vocabulary, 13 trained musicians, and known word boundaries. Leave-one-subject-out testing checks generalization across subjects, but not to unseen stimuli or continuous speech. The reported results use GPT-2; the authors also observe the same trend with SmolLM, suggesting the effect is not specific to one language model.

The imagined condition is also not raw imagined MEG. It uses an imagined-to-listened mapping trained without the held-out subject, and imagined results are averaged across six mapping variants. That design keeps the pipeline subject-independent, but it means the imagined numbers describe mapped imagined responses rather than decoding imagination directly.

Where the language prior earns its weight

The paper's practical priority follows from the strength of the neural evidence rather than from a fixed recipe. When neural predictions are strong, as in listened MEG, MEG-only ranking already beats language-model-only ranking, and the best fusion keeps a smaller language-model weight. When neural evidence is weaker, as in mapped imagined MEG, the optimum moves toward the language model and the gain over neural-only decoding is larger.

That is the comparison worth carrying forward for imagined-speech brain-computer interfaces: the value of a language prior is conditional on how reliable the neural signal is, and the paper's evidence places imagined speech on the side where a larger language contribution pays off.

Sources used in this answer