TLDR: A new research paper introduces CA-MER, a benchmark to evaluate how Multimodal Large Language Models (MLLMs) handle emotion conflicts where visual and audio cues are inconsistent. The study found that current MLLMs often over-rely on audio. To address this, the researchers propose MoSEAR, a framework with Modality-Specific Experts (MoSE) and Attention Reallocation (AR), which significantly improves MLLMs’ ability to accurately reason about emotions, especially in conflicting scenarios, without sacrificing performance on consistent emotional expressions.
Understanding human emotions is a cornerstone for effective human-computer interaction, paving the way for advanced applications in areas like educational assistance and psychological counseling. Traditionally, emotion recognition focused on single inputs or closed categories, often lacking the nuanced reasoning capabilities needed for real-world scenarios. The advent of Multimodal Large Language Models (MLLMs) has brought significant progress, allowing AI to process and interpret information from various sources like video, audio, and text, leading to more open-ended and interpretable emotion predictions.
The Challenge of Emotion Conflicts
Despite their impressive capabilities, current emotion-focused MLLMs often falter when faced with emotion conflicts. These are common scenarios where emotional cues from different modalities are inconsistent. For instance, a person might display a disappointed facial expression while speaking in a deliberately neutral tone. Existing benchmarks and models frequently overlook or even intentionally avoid such inconsistent samples, which is a significant limitation given that humans naturally express emotions inconsistently due to social norms, emotional regulation, or unconscious leakage.
Introducing CA-MER: A New Benchmark
To address this critical gap, researchers have introduced CA-MER (Conflict-Aware Multimodal Emotion Reasoning), a novel benchmark specifically designed to evaluate MLLMs under realistic emotion conflicts. CA-MER comprises three distinct subsets: video-aligned, audio-aligned, and consistent. In the video-aligned and audio-aligned subsets, only one modality (video or audio, respectively) reflects the true emotion, while others present conflicting cues. The consistent subset, conversely, includes samples where all modalities uniformly express the true emotion.
Evaluations on CA-MER revealed a systematic issue: state-of-the-art emotion MLLMs tend to over-rely on audio signals during emotion conflicts, often neglecting crucial visual cues. For example, a leading MLLM showed a substantial performance drop on video-aligned samples compared to audio-aligned ones. This audio bias was further confirmed by analyzing the models’ internal attention patterns, which showed a disproportionate focus on audio tokens. A key contributing factor identified for this bias is the extreme imbalance in the number of video and audio tokens processed by these models, with video tokens often outnumbering audio tokens by a significant margin.
Addressing the Bias with MoSEAR
To mitigate this modality bias and promote a more balanced integration of information, the researchers propose MoSEAR (Modality-Specific Experts and Attention Reallocation). This parameter-efficient framework consists of two complementary modules:
-
Modality-Specific Experts (MoSE): This module addresses bias in the fine-tuning heads of MLLMs. It uses a mixture of specialized LoRA (Low-Rank Adaptation) modules for visual, non-visual (audio and text), and omni (all modalities) inputs. A regularized gating mechanism dynamically adjusts the contributions of these experts, preventing over-reliance on any single modality during training.
-
Attention Reallocation (AR): This mechanism operates during inference to reduce bias within the frozen backbones of the MLLMs. Unlike simpler approaches that might statically shift attention, AR intelligently identifies specific attention heads that excessively focus on the audio modality on a per-sample basis. It then reallocates a portion of this attention towards visual tokens, ensuring that gains on video-aligned data do not compromise performance on audio-aligned data, and crucially, it improves performance on consistent samples as well.
Also Read:
- Advancing Emotion Understanding with Multimodal AI: A Deep Dive into Language Models
- DRKF: Advancing Emotion Recognition Through Decoupled Representations and Knowledge Fusion
Demonstrated Effectiveness
Extensive experiments across multiple benchmarks, including MER2023, EMER, DFEW, and the newly introduced CA-MER, demonstrate that MoSEAR achieves state-of-the-art performance. It shows notable improvements, particularly under modality conflict conditions, significantly reducing the performance gap between video-aligned and audio-aligned scenarios. Furthermore, MoSEAR enhances overall multimodal emotion reasoning capabilities, even in consistent emotional scenarios, without incurring a trade-off between audio and visual modalities. Human evaluations also confirmed that MoSEAR’s outputs are more consistent with human emotion understanding.
This research offers a systematic study into the challenges of emotion conflicts in MLLMs and provides an effective solution to bridge the gap, leading to more robust and accurate AI systems for understanding complex human emotions. For more in-depth details, you can read the full research paper here.
