TLDR: The MVRS dataset is a novel multimodal dataset for emotion recognition, utilizing Virtual Reality (VR) videos to elicit naturalistic emotional responses. It collects synchronized eye-tracking, body motion, ElectroMyoGraphy (EMG), and Galvanic Skin Response (GSR) data from 13 participants across five emotions (relaxation, fear, stress, sadness, joy). The research demonstrates that combining these multimodal signals, particularly through early fusion techniques and ensemble classifiers like XGBoost, significantly enhances emotion recognition accuracy and reliability compared to using single modalities.
Understanding human emotions automatically has become increasingly important in recent years, especially with the rapid advancements in artificial intelligence. Personalized emotion recognition systems hold significant promise across various fields, including healthcare, education, retail, and automotive. However, a major challenge in developing these systems is the scarcity of high-quality, multimodal data, particularly for modalities like body motion and physiological signals. Furthermore, the way emotions are triggered in participants is crucial; it needs to closely mimic real-life emotional experiences to ensure the data is truly representative.
To address these challenges, researchers Seyed Muhammad Hossein Mousavi and Atiye Ilanloo have introduced a novel dataset called MVRS: The Multimodal Virtual Reality Stimuli-based Emotion Recognition Dataset. This dataset aims to provide a rich and diverse resource for advancing emotion recognition research, particularly by leveraging the immersive capabilities of Virtual Reality (VR).
What is the MVRS Dataset?
The MVRS dataset is designed to capture naturalistic emotional responses using VR videos. It includes data from 13 participants, ranging in age from 12 to 60, and covers both genders. The emotions targeted in the dataset are relaxation, fear, stress, sadness, and joy. The data collection took place in Tehran, Iran, following strict protocols to ensure consistency and participant comfort.
The dataset is multimodal, meaning it collects several types of physiological and behavioral data simultaneously and with synchronized timestamps. These modalities include:
- Eye Tracking: Recorded using a Full High-Definition (FHD) webcam manually integrated into the VR Head-Mounted Display (HMD). This captures gaze direction, fixation duration, saccades, and pupil dilation, which are subtle indicators of cognitive and emotional states.
- Body Motion: Captured using a Microsoft Kinect version 2 sensor, which tracks skeletal joint positions and movements in 3D space. Body movements, posture, and gestures provide rich nonverbal cues about emotional states.
- ElectroMyoGraphy (EMG): Measures the electrical activity produced by skeletal muscles, effective for detecting muscle tension related to emotions like stress or fear.
- Galvanic Skin Response (GSR): Also known as ElectroDermal Activity (EDA), this tracks changes in skin conductance due to sweat gland activity, serving as a reliable marker of autonomic arousal.
These physiological signals (EMG and GSR) are recorded using an Arduino UNO board, ensuring low cost and ease of use.
Eliciting Emotions with VR
A key innovation of MVRS is its use of VR stimuli to evoke genuine emotional responses. The experiment involves a carefully designed sequence of VR videos. For instance, fear is elicited through scenes from video games and movies featuring creepy creatures or horrifying transformations. Stress is induced by clips from thrillers, like a character in a life-threatening trap with a ticking clock, or a roller coaster experience. Sadness is evoked by emotionally impactful movie scenes, such as a father sacrificing himself for his children. Joy is brought about by humorous clips, like a Mr. Bean sketch. Short relaxation videos are strategically placed between intense emotional stimuli to help participants reset their emotional state and minimize carryover effects, ensuring cleaner, independent emotional responses.
The Power of Multimodal Fusion
The research emphasizes the importance of combining these different data streams. Emotions are complex, involving intertwined physiological, behavioral, and cognitive processes. Relying on a single modality often provides an incomplete picture. By integrating data from body motion, physiological signals, and eye-tracking, multimodal fusion techniques can capture richer representations of affective dynamics, improving robustness against noise and individual variability.
The study explored three multimodal fusion approaches: two early-stage fusion methods (feature selection and autoencoder-based dimensionality reduction) and one late-stage fusion method (majority voting). Early-stage fusion combines features before classification, allowing the model to learn complex inter-modal relationships. Late-stage fusion processes each modality independently and combines their final predictions.
Also Read:
- Understanding Student Engagement: How Biosensors and Multimodal Analytics are Shaping Online Education
- EHWGesture: A Comprehensive Resource for Understanding Clinical Hand Movements
Key Findings and Contributions
The evaluation of the MVRS dataset yielded several significant insights:
- Effectiveness of VR: VR environments proved highly effective in eliciting naturalistic emotional responses, particularly for strong emotions like stress and fear, which showed clearer separability.
- Superiority of Multimodal Data: Fused datasets consistently outperformed unimodal baselines. Combining eye-tracking, body motion, EMG, and GSR data significantly enhanced emotion recognition accuracy and reliability.
- Early Fusion Excels: Early-stage fusion, especially when combined with autoencoder-based dimensionality reduction, achieved the highest classification performance, demonstrating its ability to create more structured and discriminative feature spaces.
- XGBoost as a Top Performer: Among the classifiers tested (Random Forest, Decision Tree, XGBoost), XGBoost consistently delivered the best results across both unimodal and multimodal datasets.
- Discriminative Features: The study identified the most influential features for each modality: temporal and kinematic descriptors for body motion, skin conductance measures for EMG+GSR, and pupil and gaze-based metrics for eye tracking.
The MVRS dataset, along with its associated code, is publicly available on platforms like Kaggle, Zenodo, Figshare, and IEEE Data Port. This resource provides a valuable foundation for researchers to develop more accurate, robust, and naturalistic emotion recognition systems, with potential applications in healthcare, education, and adaptive human-computer interaction.
For more detailed information, you can access the full research paper here.
