EEGAIN: Resolving the Reproducibility Crisis in EEG-based Emotion Recognition

TL;DR

Despite the explosion of interest in Electroencephalography-based Emotion Recognition (EEG-ER), researchers have historically been "comparing apples to oranges." This paper uncovers deep inconsistencies across 216 major studies and introduces EEGAIN, an open-source framework designed to unify data loading, pre-processing, and evaluation metrics. Their findings are sobering: many "state-of-the-art" models barely outperform random chance in subject-independent scenarios once evaluation is strictly standardized.

The Chaos in the Current SOTA

The core motivation behind this work is the observation of 5 critical barriers to progress in EEG-ER:

1. Dataset Bias: Most research (90%) uses only 1 or 2 datasets, which are often too small (<100 participants).
2. Pre-processing Variability: Results aren't comparable if one researcher uses a 4-45Hz band-pass filter and another uses 1-64Hz.
3. Data Splitting Inconsistency: The choice between Subject-Dependent (easier) and Subject-Independent (harder) splits fundamentally changes the task difficulty.
4. The Ground-Truth Threshold Trap: Binarizing continuous valence/arousal ratings (1-9) using 4.0 vs 5.0 shifts the class distribution, potentially inflating accuracy via class imbalance.
5. Metric Misuse: A heavy reliance on Accuracy without reporting F1-scores or standard deviations masks model instability.

Methodology: The EEGAIN Architecture

The authors propose a modular Python-based framework to enforce rigor. The architecture consists of five functional blocks:

• Data: One-line loaders for AMIGOS, DEAP, DREAMER, MAHNOB-HCI, SEED, and SEED-IV.
• Transform: Standardized Butterworth filtering, artifact removal, and windowing.
• Split: Native support for LOSO (Leave-One-Subject-Out) and LOTO (Leave-One-Trial-Out).
• Model: Pre-integrated SOTA architectures: EEGNet, DeepConvNet, ShallowConvNet, and TSception.
• Logger: Automated TensorBoard integration for tracking MCC, Kappa, and F1-weighted.

A critical design choice was the inclusion of Trivial Baselines. By providing a "Majority Class" classifier, the framework forces researchers to prove their Deep Learning model is actually learning neural patterns rather than just exploiting a skewed dataset.

Experimental Revelation: Is SOTA Real?

The authors conducted a massive benchmark across 6 datasets in the challenging LOSO scenario.

The results provide a reality check:

• Categorical Success: On SEED and SEED-IV (Discrete emotions like Happy/Sad), models like DeepConvNet showed substantial gains (Weighted F1: 0.52 vs. baseline 0.34).
• Dimensional Struggles: On datasets like DEAP and DREAMER (Valence/Arousal), most models offered "marginal if non-existent" improvements over a trivial majority-class classifier.

Critical Insights & Future Outlook

The "Subject Gap" remains the biggest hurdle. High inter-subject variability in EEG means a model trained on Subject A likely fails on Subject B.

Takeaways for the community:

• Standardize Thresholds: The authors propose a fixed threshold (≤ 3.0 for DREAMER; ≤ 4.5 for 9-point scales) to keep metrics honest.
• Beyond Accuracy: You must report F1-scores and Standard Deviation. If Accuracy is 70% but F1 is 40% on an imbalanced set, the model is failing.
• Reproducibility First: Using a framework like EEGAIN (or the subsequent LibEER) is no longer optional; it is a prerequisite for scientific validity.

EEGAIN essentially provides the "ImageNet-style" rigor that the physiological signal processing community has desperately needed to move from artisanal experiments to industrial-grade AI.

Reference: Kukhilava, N., et al. "Evaluation in EEG Emotion Recognition: State-of-the-Art Review and Unified Framework." (2025/2026 Update).