Can EEG-guided latent generative modeling deliver high-fidelity target speaker extraction?

Masking caps fidelity where voices overlap

Target speaker extraction (TSE) takes a mixture of several talkers and returns only the voice the listener wants. Most prior neuro-steered systems do this by estimating a time-frequency mask: a filter applied to the mixture's spectrogram that suppresses the interfering voices. The paper argues this design has two costs. In regions where target and interferer overlap in both time and frequency, the mask cannot cleanly separate them, so fidelity drops. And masking systems are often limited to narrowband audio, which caps how much high-frequency detail survives.

NeuroCodec changes the object being predicted. Instead of filtering the mixture, it predicts the target speaker's latent representation inside a frozen neural audio codec, the DAC encoder and decoder. The mixture and the EEG are inputs; the output is a set of latents that the frozen DAC decoder turns back into a waveform. The paper frames this as a shift from suppressing interference in the mixture domain to aligning cross-modal representations, letting the pretrained codec supply acoustic priors while the trainable network concentrates on reading attention from EEG.

EEG steers the latents through attention and Mamba

The pipeline has four stages. A frozen DAC encoder compresses the mixture waveform into latents with dimension 1024. An EEG encoder, adapted from M3ANet, applies third-order Chebyshev graph convolutions across electrodes to capture spatial dependencies, then parallel 1-D convolutions with kernel sizes 3, 7 and 11 to capture multi-scale temporal structure. A cross-modal block then lets the acoustic latents act as queries while the EEG features act as keys and values in multi-head cross-attention, so neural activity selects which acoustic content to keep. A Mamba block follows to model long-range temporal dependencies in linear time.

One design choice matters for cost. Earlier systems upsampled EEG toward the audio rate; NeuroCodec instead operates at the codec latent rate, 50 Hz or 86 Hz depending on the dataset, close to the native 128 Hz EEG sampling. That avoids the initial upsampling step and reduces interpolation and computation. The predicted latents are quantized with the frozen DAC codebooks and decoded into the output waveform.

A six-phase curriculum builds spectral fidelity

Latent reconstruction alone is not enough. The ablation on the KUL dataset shows the progression clearly: with only the latent reconstruction loss, the model reaches STOI 0.7112, ESTOI 0.5734 and PESQ 1.5030. Adding the multi-scale log-Mel spectral loss raises these to 0.8015, 0.6861 and 1.9204. Adding the adversarial feature-matching loss instead gives 0.8242, 0.7122 and 2.1032. Combining all three objectives gives the best result: 0.8681, 0.7544 and 2.5623.

The paper describes a six-phase training curriculum that introduces latent reconstruction, multi-scale spectral and adversarial objectives progressively. The spectral loss compares log-Mel spectrograms at seven scales up to 20 kHz for Cocktail Party and four scales up to 4 kHz for KUL, matching each dataset's bandwidth. The adversarial side uses multi-period discriminators for waveform periodicity and a multi-band multi-scale STFT discriminator for spectral consistency. These are training choices, not inference-time additions, and the ablation attributes the perceptual gain mainly to the spectral and adversarial terms.

Objective gains over M3ANet are real but modest

On the Cocktail Party dataset, all models were evaluated under the same subject-independent split, with test subjects unseen during training, and outputs were resampled to 8 kHz before scoring so that NeuroCodec's wider bandwidth would not give it an inherent advantage. Under that setup, NeuroCodec reaches STOI 0.8273, ESTOI 0.721 and PESQ 2.31, against M3ANet's 0.812, 0.725 and 2.23 and BASEN's 0.790, 0.715 and 1.91.

Read carefully, the pattern is mixed. NeuroCodec leads on STOI and PESQ but is slightly below M3ANet on ESTOI, 0.721 versus 0.725. The paper describes its position as strong on intelligibility and perceptual quality while remaining highly competitive in ESTOI. On the KUL dataset, compared with NeuroSpex+, NeuroCodec improves STOI from 0.859 to 0.868 and PESQ from 2.514 to 2.562. These are small margins, and the KUL baseline scores come from the NeuroSpex+ paper under the same subject-independent configuration rather than from re-running every model.

Listeners and bandwidth favor the generative approach

The objective metrics do not capture everything. A listening experiment with 20 participants rated NeuroCodec higher for perceptual accuracy, with a mean score of 82.24 compared with 49.63 for M3ANet. That is the largest gap reported in the paper, and it points at exactly the quality dimension the masking critique targets.

Spectrogram comparison supports the same reading. M3ANet shows high-frequency truncation around 7 kHz, while NeuroCodec reconstructs spectral content up to 20 kHz with clearer harmonic structure, and in overlapping-speech regions it shows cleaner target-speaker separation. The paper links this to direct generation of target latents rather than masking the mixture. The 44.1 kHz reconstruction is a property of the Cocktail Party setup, which uses the 44.1 kHz DAC encoder; the KUL setup instead upsamples 8 kHz audio to 16 kHz, so the wideband result should not be assumed for that dataset.

EEG guidance and Mamba carry most of the gain

The ablation isolates which components matter. Replacing EEG with Gaussian noise drops performance sharply, to STOI 0.5412, ESTOI 0.3921 and PESQ 0.7845, which the authors read as evidence that the model depends on neural guidance rather than learning extraction from the mixture alone. Removing the graph convolution spatial mapping gives 0.6725, 0.5312 and 1.2840; removing the multi-scale 1-D convolutions gives 0.6854, 0.5408 and 1.3912. Both fall below the full encoder's 0.7112, 0.5734 and 1.5030, so cross-channel spatial topology and multi-scale temporal receptive fields both contribute.

Temporal modeling also separates the options. A standard Transformer reaches STOI 0.6120, ESTOI 0.4703 and PESQ 0.8702; adding Snake activations improves all three to 0.6480, 0.5162 and 1.0340; the causal Mamba reaches 0.7112, 0.5734 and 1.5030. These ablations were run on KUL with the reconstruction loss only, so they compare architectures under a weaker objective than the final model, not the full system.

Streaming works, but the evidence has edges

The paper extends the method to OnlineNeuroCodec, a causal streaming formulation for real-time use. It conditions each hop on previous target-latent estimates through an auditory attractor: a linear layer turns the mean of past latents into scale and shift parameters applied to the current mixture latents before EEG cross-attention. Unlike NeuroHeed, which re-encodes previous waveform outputs with a dedicated LSTM speaker encoder, this updates the speaker state directly in codec latent space with a parameter-free exponential moving average. Training uses speaker dropout with probability 0.2 so the model still works when history is unavailable, and gradients are stopped at the stored state between hops.

Scope conditions matter for reading the headline numbers. All experiments use subject-independent evaluation, so the results describe unseen test subjects rather than per-listener calibration. Window sizes differ by dataset, 2-second non-overlapping windows for Cocktail Party and 4-second windows with a 1-second hop for KUL, chosen to match prior work on each dataset. NeuroHeed+ and MSFNet are absent from the comparison table because no public code allowed re-evaluation under the same subject-independent setting. The paper reports the streaming extension as a formulation and basis for hearing assistive devices, not as a validated real-time deployment.

Where the generative route earns its place

The paper's most decision-relevant split is between objective scores and perceived quality. On 8 kHz-resampled objective metrics, NeuroCodec's advantage over M3ANet is narrow and not uniform: it wins STOI and PESQ but trails slightly on ESTOI. The gap widens sharply in the 20-participant listening test, 82.24 versus 49.63, and in the spectrogram evidence above 7 kHz. For a reader deciding whether latent generative modeling is worth the added complexity, the paper's answer is that the payoff shows up in bandwidth and perceived fidelity rather than in large objective-metric margins.

That makes the choice scenario-dependent. If the target is a narrowband objective benchmark, the improvement over strong masking baselines is incremental. If the target is wideband, natural-sounding output for a neuro-steered hearing device, the combination of 44.1 kHz reconstruction, preserved content up to 20 kHz and the listening-test margin is the paper's central case, with the streaming formulation offered as the path toward real-time use rather than a demonstrated one.

Sources used in this answer