TLDR: The ECMF framework addresses challenges in multimodal emotion recognition, particularly data scarcity and noisy labels. It leverages large-scale pre-trained models for visual (dual-branch encoder), audio (HuBERT-Large), and textual (context-enriched with LLMs) feature extraction. A self-attention-based fusion strategy dynamically integrates these features, while a multi-source labeling strategy refines noisy training data. The framework significantly outperforms the baseline on the MER2025-SEMI dataset, achieving an 87.49% weighted F-score.
Understanding human emotions is a crucial step in making artificial intelligence systems more interactive and human-like. This capability is vital for improving how we interact with computers and enhancing user experiences. The MER2025 competition’s MER-SEMI challenge focuses on advancing emotion recognition, particularly in a semi-supervised learning environment where both labeled and unlabeled video data are used. The goal is to classify video samples into six emotion categories: worry, happiness, neutral, anger, surprise, and sadness.
However, recognizing emotions from video presents significant challenges. Videos involve multiple types of data—text, visual, and audio—and effectively extracting and integrating information from these diverse sources is complex. Additionally, the scarcity of labeled data makes it difficult to train robust models, increasing the reliance on methods that can learn from less supervision or leverage pre-trained knowledge.
To tackle these issues, researchers have proposed a novel framework called ECMF, which stands for Enhanced Cross-Modal Fusion. This framework aims to improve multimodal emotion recognition by leveraging large-scale pre-trained models to extract rich features from visual, audio, and textual data. It also introduces a sophisticated fusion strategy and a method to refine noisy labels in the training data.
Extracting Rich Emotional Cues from Multiple Sources
The ECMF framework begins by extracting features from each modality. For visual information, it uses a dual-branch visual encoder. This innovative approach captures both broad, overall scene features from video frames and detailed, localized facial expressions. This ensures that the system doesn’t miss important visual cues, whether they are subtle facial movements or larger body language signals.
For textual content, the framework employs a context-enriched method. Recognizing that text alone might not always convey strong emotional signals, this method uses advanced large language models (LLMs) like GPT-4 and Qwen-Omni. These LLMs help to enrich the emotional context of the input text by generating pseudo-labels, emotion-related keywords, and detailed video descriptions. This augmentation helps the system better understand the emotional nuances embedded in the text.
When it comes to audio, the framework utilizes specific layers from the HuBERT-Large model. These layers are chosen because they are particularly effective at capturing prosodic and spectral patterns, which are crucial for discerning emotions from speech, such as variations in tone, pitch, and rhythm.
Intelligent Fusion for Comprehensive Understanding
After extracting features from each modality, the next critical step is to effectively combine them. The ECMF framework proposes a sophisticated fusion strategy based on self-attention mechanisms and residual connections. Instead of simply combining features, this method dynamically weighs the importance of each modality. This is crucial because different modalities might contribute differently to emotion recognition in various contexts, and some might even contain irrelevant information. By using self-attention, the system can focus on the most relevant emotional cues across modalities, while residual connections help preserve the original, important information.
A unique aspect of this fusion is the use of a learnable ‘Modal_Token’ for each modality. Similar to how positional encodings work in Transformer models, these tokens help the system understand which features belong to which modality (audio, text, or video), allowing for a more informed integration process.
Enhancing Data Quality and Reliability
Beyond its architectural design, the ECMF framework also addresses a common challenge in real-world datasets: noisy labels. The researchers observed inconsistencies in some training labels within the MER2025-SEMI dataset. To correct this, they implemented a multi-source labeling strategy. This involves training ‘weak classifiers’ on each individual modality and also leveraging LLMs like Qwen-Omni to generate auxiliary emotion labels. A majority voting scheme across all these sources is then used to derive more accurate, refined labels. For highly inconsistent predictions, manual review and correction are performed to ensure high-quality data.
Furthermore, the framework employs ensemble learning to boost the reliability of the final emotion predictions. By training several variations of the model and combining their predictions through majority voting, the system achieves more robust and accurate results.
Also Read:
- SpeakerLM: An End-to-End AI Solution for Speaker Diarization and Recognition
- Enhancing Recommendations with Semantic Item Graphs and Noise Robustness
Impressive Performance Gains
Experiments conducted on the official MER2025-SEMI dataset demonstrated that the ECMF framework significantly outperforms the official baseline. The proposed method achieved a weighted F-score of 87.49% on the test set, a substantial improvement compared to the baseline’s 78.63%. This validates the effectiveness of the framework’s various components, from the refined feature extraction techniques to the intelligent fusion strategy and the data quality improvements.
In summary, the ECMF framework offers a comprehensive solution for multimodal emotion recognition, particularly in scenarios with limited labeled data. By intelligently combining pre-trained models, advanced fusion techniques, and robust label refinement strategies, it paves the way for more accurate and reliable AI systems capable of understanding human emotions. For more details, you can refer to the full research paper.
