Can EEG and EOG fusion improve auditory attention detection reliability?

Why EEG alone leaves attention underdetermined

Auditory attention decoding (AAD) asks a machine to infer which speaker a listener is following from physiological signals, so that a hearing device or interface can act on that inference. The paper treats EEG as the dominant modality for this task because it reflects cortical activity tied to attentional selection, but it also lists the weaknesses that matter in realistic audio-visual scenes: low signal-to-noise ratio, high susceptibility to artifacts, and poor generalization across subjects and sessions.

The paper's argument for adding EOG is that eye movements carry behavioral information about auditory attention, and EOG records ocular activity directly. It cites prior findings that gaze follows attended natural speech and that visual speech cues enhance neural tracking of the attended speaker. The same literature also warns that ocular activity can introduce confounding patterns and bias spatial-attention decoding, which is why the authors argue EOG should be modeled explicitly and integrated adaptively rather than discarded as noise or merged by simple concatenation.

Three gaps the network is built to close

The paper frames existing EEG-EOG fusion for AAD as limited in three specific ways. First, continuous cross-modal dependencies are insufficiently modeled: cross-attention has worked in related physiological and multimodal tasks, but bidirectional token-wise interaction between EEG and EOG remains underexplored in AAD. Second, temporal modeling is inefficient, because convolutions emphasize local patterns while self-attention is computationally expensive on long sequences. Third, modality quality varies from sample to sample due to artifacts, subject state, and ocular dynamics, so fixed weighting and direct concatenation cannot adapt.

Each component of RAMamba-Net maps onto one of these gaps. Mamba, a selective state-space model with linear-time complexity, addresses the temporal-cost problem. Cross-modal attention addresses the interaction problem. A reliability-aware module addresses sample-wise variation by estimating how much to trust each modality in each individual sample.

Separate encoders preserve what each signal measures

The EEG encoder builds on a band-aware convolutional Transformer and adds Mamba blocks. For each of K EEG frequency bands, a front Mamba block reinforces the temporal structure of the raw band-specific sequence before a band-specific stem extracts local spectro-temporal features; a middle Mamba block then enriches high-level feature dynamics before the Transformer encoder. The band representations are concatenated and passed through a frequency-aware attention module that adaptively determines each band's contribution and remaps the result into a unified EEG token space.

The EOG encoder uses two parallel branches: a temporal Transformer for fine-grained temporal dependencies and a spatial Transformer for inter-channel dependencies. Their outputs are concatenated along the feature dimension and projected into a shared embedding space. The stated purpose of this dual-branch design is to keep EOG's temporal and spatial inductive biases while producing tokens dimensionally aligned with the EEG tokens, which is a precondition for the token-wise interaction that follows.

Attention exchanges evidence before weights decide trust

Cross-modal attention makes the interaction explicit. For each modality, its tokens act as queries while the other modality supplies keys and values, so each stream retrieves complementary information from its partner. The update is applied in both directions, with residual learning, layer normalization, and a feed-forward network, so each stream keeps its modality-specific information while incorporating evidence from the other. The paper describes this symmetric design as promoting balanced information exchange and better cross-modal alignment.

The reliability-aware module (RAM) then estimates sample-wise weights. Each modality's token sequence is mean-pooled into a sample-level descriptor, with a stop-gradient that decouples reliability estimation from encoder optimization. Two consistency scores are computed: feature consistency, a normalized cosine similarity between the pooled EEG and EOG vectors, and prediction consistency, derived from the Jensen-Shannon divergence between the two unimodal predicted distributions. These descriptors and scores form a reliability representation that a small network maps to EEG and EOG weights, which modulate the cross-modally interacted features before concatenation and classification.

Training combines three losses: cross-entropy on the final multimodal prediction, cross-entropy on each modality's auxiliary prediction to keep each encoder discriminative, and an auxiliary loss on each EEG frequency band. The paper reports that the reliability-aware module suppresses unreliable modality evidence and is robust to signal perturbation and parameter variation.

The 5.76% gain compares fusion against single modalities

The headline number is an accuracy gain of 5.76% over unimodal baselines, measured on two AAD benchmarks. The comparison is between the fused EEG-EOG model and baselines that use one modality, so it answers whether combining the two signals helps, not whether RAMamba-Net beats every other multimodal design by that margin. The paper also reports more robust decoding and more discriminative representations, and its analyses attribute the improvement to two separate mechanisms: explicit cross-modal interaction improving multimodal alignment, and the reliability-aware module suppressing unreliable modality evidence.

That separation matters for reading the result. The 5.76% figure is a single aggregate over the benchmarks, while the mechanism claims come from further analyses of alignment and of behavior under perturbation and parameter variation. A reader should not treat the accuracy gain as proof that either component alone produces the full effect, because the paper's ablation-level attribution is described qualitatively rather than as a per-component accuracy number in the available text.

Benchmarks, perturbation tests, and the generalization gap

The evidence rests on two AAD benchmarks, named in the discussion as AVGC and DTU. Robustness claims are supported by tests under signal perturbation and parameter variation, which probe sensitivity to degraded inputs and to model settings rather than to new subjects or recording setups. The paper's own discussion places cross-subject and cross-dataset generalization under more challenging distribution shifts in future work, alongside efficient adaptation and deployment to reduce computational cost.

This boundary is the one most likely to change how a reader weighs the result. The introduction motivates the work with artifacts, subject state, and ocular dynamics varying across samples, and the reliability module is designed for exactly that variation, but the reported evaluation does not establish that the sample-wise weighting transfers to unseen users or recording conditions. The gain is therefore best read as evidence that adaptive EEG-EOG fusion helps within the tested benchmarks, not as a claim about clinical or consumer deployment.

EOG earns its place only when it is weighted

The paper's most transferable judgment is about how to add EOG, not whether to add it. Its introduction states that ocular activity can bias spatial-attention decoding, and its design response is to model EOG with dedicated temporal and spatial branches and to let a sample-wise reliability estimate decide how much each modality contributes. The reported robustness to signal perturbation and parameter variation is the evidence offered for that choice, and the reported alignment improvement is the evidence offered for explicit bidirectional attention over weaker interaction.

For a reader deciding what to take from this work, the priority is the pairing: cross-modal attention supplies the complementary evidence, and the reliability weights decide when that evidence should be trusted. The 5.76% gain over unimodal baselines is the number to carry forward, with the caveat that it was obtained on two benchmarks and that cross-subject and cross-dataset behavior remains untested in this paper.

Sources used in this answer