TLDR: LibEMER is an open-source PyTorch library and benchmark addressing the lack of standardization and reproducibility in EEG-based multimodal emotion recognition (EMER). It provides unified protocols for data processing, model implementation, and evaluation across three public datasets and two learning tasks, offering a fair framework for assessing deep learning methods and guiding future research.
The field of EEG-based multimodal emotion recognition (EMER), which focuses on identifying human emotional states using brain activity and other physiological signals, has seen significant advancements. However, it has faced notable challenges, including a lack of open-source implementations, standardized benchmarks for fair comparison, and in-depth discussions on key issues and future directions.
To address these critical limitations, researchers have introduced LibEMER, a novel and unified evaluation framework. LibEMER provides fully reproducible PyTorch implementations of carefully selected deep learning methods. It also establishes standardized protocols for every stage of the research process, from data preprocessing and model realization to experimental setups. This comprehensive framework allows for unbiased performance assessment across three widely-used public datasets and two distinct learning tasks.
Key Contributions of LibEMER
LibEMER makes several important contributions to the EMER community:
-
It offers an open-source multimodal deep learning library specifically designed for emotion recognition, integrating EEG with other physiological signals. This library provides a standardized, end-to-end workflow, simplifying data loading, training, and evaluation for researchers.
-
It establishes a fair, transparent, and standardized benchmark for multimodal EEG-based emotion recognition. This benchmark includes meticulously defined protocols for data preprocessing, experimental tasks, and evaluation metrics, ensuring consistent and unbiased assessment of various methods.
-
The framework facilitates extensive experiments on implemented methods, strictly adhering to the proposed benchmark. This leads to thorough analyses of results, systematically reporting model performance across different tasks and datasets, and offering valuable insights to guide future research.
Building the Benchmark
The LibEMER benchmark is built upon several key components:
-
Methods Selection: A systematic approach was used to curate 21 representative deep learning methods from recent publications, categorized into architectural paradigms like DNN, CNN, RNN, Transformer, and GNN. These methods were rigorously reimplemented to ensure accuracy.
-
Datasets: Three widely adopted public datasets—SEED, SEEDV, and DEAP—were chosen based on usage frequency, sample scale, and modality coverage. SEED and SEEDV include EEG and eye movement data, while DEAP encompasses a broader range of physiological signals such as EEG, EOG, EMG, GSR, and respiration.
-
Data Preprocessing: A unified preprocessing framework was developed. For EEG, this involves baseline removal, band-pass filtering, artifact mitigation using PCA, and extraction of differential entropy (DE) features. Eye movement data involves extracting 33 features, and peripheral physiological signals from DEAP are processed to extract statistical features, ensuring temporal alignment across modalities.
-
Data Splitting: To prevent implicit model tuning and ensure accurate generalization assessment, a strict cross-validation strategy is employed. Data is split into training, validation, and test sets in a 3:1:1 ratio, with no overlap. Model selection is based on optimal performance on the validation set.
-
Evaluation and Metrics: Models are evaluated using two primary tasks: subject-dependent (training and testing on different trials of the same subject) and subject-independent (training and testing on different subjects). The F1-score is the primary metric, balanced for precision and recall, with accuracy also reported.
Also Read:
- Advancing Brain-Computer Interfaces with Rapid Online Adaptation
- Understanding AI’s Emotional Leap: Privacy, Ethics, and the Future of Affective Computing
Experimental Insights and Analysis
Experiments conducted with LibEMER revealed several important observations:
-
Reproduced models often showed a performance decline compared to their original reported values, though within a 10% margin. This was attributed to insufficient detail in original publications and disparities in hardware/software environments.
-
Models generally performed better on datasets with objective, discrete labels (like SEED) compared to those with subjective ratings on continuous dimensions (like DEAP). This suggests that physiological signals align more closely with objective responses.
-
The strategic design of learning tasks and fusion strategies, tailored to the unique characteristics of different modalities, proved more critical for performance enhancement than the choice of model architecture alone. DNN-based methods, for instance, demonstrated robust performance due to their well-designed strategies for cross-modal representation.
-
A significant performance discrepancy and high instability (large standard deviations) were observed under the unified benchmark. This highlights that the limited size of mainstream datasets hinders deep learning models from effectively learning complex cross-modal emotion representations, underscoring the need for larger, high-quality datasets.
In conclusion, LibEMER aims to accelerate the standardization and transparency of EEG-based multimodal emotion recognition research, facilitating the development of new algorithms and providing valuable insights into current challenges and promising future directions. You can find more details about this work in the full research paper: LibEMER: A Novel Benchmark and Algorithms Library for EEG-Based Multimodal Emotion Recognition.
