Can contrastive EEG-text alignment decode imagined speech beyond closed-set labels?

A label classifier becomes a similarity matcher

The paper's starting complaint is specific. Conventional imagined-speech decoders treat each semantic category as an independent label, so the output is a single class index and nothing about how intents relate to one another. The authors instead train an EEG encoder to place each trial inside the embedding space of a frozen pretrained text encoder, so that a trial and a sentence become comparable vectors.

The target space is fixed. Text embeddings are computed from the original Chinese stimulus sentences and Chinese category names, L2-normalized, and the text encoder's weights are never updated during EEG training. The authors justify this by the small Chisco vocabulary: fine-tuning the language model could distort pretrained semantic structure, reduce separability between classes, and add overfitting risk, while a frozen encoder avoids a moving alignment target.

The EEG side is trainable. Each trial has 122 channels and 1651 time samples; an EEGNet-based convolutional backbone extracts spatiotemporal features, a four-layer Transformer with eight attention heads models temporal structure, and attention pooling plus a small MLP projects the result into a 512-dimensional, L2-normalized vector. Because both sides sit on a unit hypersphere, cosine similarity is the natural comparison.

Two losses trade semantic structure against separability

A pure alignment objective would pull EEG toward sentence meaning but might blur categories that are close in language space. The composite loss therefore combines a semantically reweighted EEG-sentence alignment term (a SoftNCE-style contrastive loss) with a text-prototype-based cross-entropy term for class discrimination. The first encourages the geometry of language; the second preserves separability among the 39 Chisco intent categories.

Category prediction at test time follows directly from that design. Each EEG embedding is compared with the frozen text-derived prototypes of the 39 categories, and the highest-similarity prototype becomes the predicted label. No separate classifier head is needed, which is why the same similarity value can later be reused as a reliability score.

Comparable accuracy under sentence-grouped folds

Evaluation uses stratified, sentence-grouped five-fold cross-validation with a fixed seed, so all trials belonging to one sentence stay in the same fold; 15% of the remaining sentence groups serve as validation, and held-out test folds are used only for final evaluation. The dataset is Chisco version 1.1.2 with five participants, and only the 3.3 s imagined-speech epochs are analyzed.

The headline comparison is that the contrastive framework achieved decoding performance comparable to a task-optimized closed-set classifier. The reported measures are Top-k accuracy, macro-F1, and balanced pairwise matching accuracy (2v2), the last estimated from 20,000 class-balanced pairs per test fold with a 50% theoretical chance level. Chance for Top-k and macro-F1 comes from a stratified random baseline reflecting each fold's training class distribution, estimated with 10,000 Monte Carlo samples per fold.

One caveat belongs with this comparison rather than at the end: the two systems were not trained identically. The contrastive model used batch size 64, learning rate 3×10⁻⁴, Gaussian noise, channel dropout, and temporal shifts, and selected checkpoints by validation Top-5; the closed-set classifier used batch size 16, learning rate 5×10⁻⁴, no extra augmentation, and selected by validation Top-1. The paper states these task-specific configurations openly, so "comparable" describes the achieved held-out results, not an identical training budget.

MaxCosine ranks which trials are worth trusting

Because the predicted category is chosen by maximum cosine similarity, every prediction already carries a score: MaxCosine, the similarity between the EEG embedding and the selected category prototype. The authors ask whether that raw score, with no extra confidence model, separates correct from incorrect Top-1 predictions.

For Subject 01, fold-specific AUROC values ranged from 0.605 to 0.678, with a mean of 0.640 ± 0.027, all above the 0.50 chance reference. Across all five subjects the mean AUROC stayed above 0.50, ranging from 0.60 to 0.64. The authors describe this as informative but incomplete reliability ranking and explicitly do not treat MaxCosine as a calibrated probability.

The more decision-relevant test is selective decoding. Thresholds were derived only from validation score distributions at target coverage levels of 100%, 80%, 60%, 40%, and 20%, then applied unchanged to held-out folds. At the 20% target, achieved test coverage stayed near the intended operating point for every subject (19.3% to 20.4%). Retaining only the highest-MaxCosine trials raised Top-1 accuracy from 8.3–10.0% at full coverage to 15.6–19.2%, and Top-5 accuracy from 29.2–32.7% to 42.9–50.5%.

That gain is a ranking effect, not a new decoder. The same model is simply allowed to abstain on most trials; the paper does not claim that the retained subset is representative of all imagined-speech trials, nor that these thresholds transfer to online use.

Semantic axes never used in training

To test whether the embedding carries structure beyond the 39 labels, the authors define five bipolar axes in the frozen language space — Practical–Emotional, Physical–Cognitive, Goal-directed–Leisure, Mobility–Personal life, and Individual–Social — and introduce them only at evaluation time. No classifier is trained on them and they are not used as supervision.

Balanced accuracy of agreement between the axis assignment derived from held-out EEG embeddings and the one derived from the corresponding sentence embeddings was above the 50% chance level on all five axes. The strongest agreement was Practical–Emotional at 61.3%, followed by Goal-directed–Leisure at 60.0% and Physical–Cognitive at 59.2%; Mobility–Personal life reached 53.3% and Individual–Social 52.9%.

The pattern matters more than the average. The three axes with the clearest agreement are also the ones whose poles are most distinct in ordinary language use, while the two weakest axes are closer to chance. The paper reports this as evidence that broader semantic organization is preserved, but the margins on the weakest axes are small enough that they should not be read as reliable decoding of those dimensions.

Unseen sentences are a visualization, not a test

The Semantic Gallery Probe compares a single held-out EEG trial against a fixed gallery of manually written sentences that were never used in training, projecting the gallery to two dimensions and coloring each sentence by its cosine similarity to the trial. In the example shown for Subject 1, a trial whose target category was Transport concentrates similarity on Transport-related sentences such as vehicle maintenance, while semantically distant regions such as Nature stay comparatively low.

The authors are explicit that this is a qualitative visualization, not a standalone performance metric and not evidence of open-vocabulary generalization. It illustrates where a trial places similarity mass in an unseen gallery; it does not quantify accuracy on unseen sentences. Similar qualitative patterns were reported across subjects, but no numeric open-vocabulary score is claimed.

Where the evidence stops

Three boundaries follow directly from the reported design. First, all results come from five native Chinese speakers using the Chisco cue-based protocol with 122 retained channels and preprocessed signals, so nothing here establishes cross-language or cross-paradigm transfer. Second, the semantic-axis analysis is described as exploratory and post hoc, computed on held-out trials but with axes chosen after training. Third, the reliability analysis is offline: thresholds were derived from validation folds and applied to held-out folds, and the paper states this does not establish probabilistic calibration or operational online performance.

The practical reading is a priority ordering rather than a replacement claim. If the goal is fixed-vocabulary intent recognition, the contrastive framework offers no demonstrated accuracy advantage over the closed-set classifier. Its distinctive value is the pair of capabilities that come free with the similarity formulation: a MaxCosine score that, at roughly 20% coverage, roughly doubled Top-1 accuracy and lifted Top-5 accuracy by about 15 to 20 percentage points across subjects, and an embedding that agrees with evaluation-only semantic axes above chance. Those are the results to weigh when deciding whether language-space alignment is worth the added machinery.

Sources used in this answer