TLDR: MME-Emotion is a new, comprehensive benchmark for evaluating the emotional intelligence of multimodal large language models (MLLMs). It uses over 6,000 video clips across eight tasks and an automated evaluation system with recognition, reasoning, and Chain-of-Thought scores. The study found that current MLLMs have unsatisfactory emotional intelligence, with top models scoring below 40% in recognition, highlighting the need for better reasoning and multimodal integration.
A new and comprehensive benchmark, MME-Emotion, has been introduced to rigorously evaluate the emotional intelligence of multimodal large language models (MLLMs). This development marks a significant stride in affective computing, a field dedicated to bridging the gap between human emotional capabilities and machine understanding.
Historically, evaluating emotional intelligence in MLLMs has faced challenges due to limited scenario coverage and inconsistent evaluation methods. This made it difficult to ascertain how well MLLMs could generalize their emotional understanding across diverse real-world situations, accurately identify emotional states, and pinpoint the underlying causes of those emotions.
MME-Emotion aims to overcome these limitations by providing a systematic framework for assessing both emotional understanding and reasoning. It stands as the largest emotional intelligence benchmark for MLLMs to date, comprising over 6,000 carefully curated video clips. These clips are paired with specific question-answering tasks, covering 27 distinct scenario types across eight emotional tasks. These tasks include emotion recognition in controlled lab settings and in the wild, emotion recognition under noise, fine-grained emotion recognition, multi-label emotion recognition, sentiment analysis, fine-grained sentiment analysis, and intent recognition.
A core innovation of MME-Emotion is its holistic evaluation suite, which employs a multi-agent system framework for automated assessment. This system utilizes an “MLLM-as-judge” strategy, where a powerful MLLM evaluates the performance of other models. It uses three unified metrics: a recognition score (Rec-S) for accuracy, a reasoning score (Rea-S) for evaluating the logical steps, and a Chain-of-Thought (CoT) score, which combines both. The reliability of this automated evaluation process has been extensively validated by human experts, demonstrating a strong correlation with human judgment.
The researchers conducted a rigorous evaluation of 20 advanced MLLMs using MME-Emotion, uncovering crucial insights into their current state of emotional intelligence. The findings indicate that the overall emotional intelligence of current MLLMs remains unsatisfactory. Even the top-performing model, Gemini-2.5-Pro, achieved only a 39.3% recognition score and a 56.0% CoT score. The average performance across all evaluated MLLMs was notably lower, underscoring the substantial room for improvement.
Interestingly, the study observed that generalist models, such as Gemini-2.5-Pro and GPT-4o, derive their emotional intelligence from their broad multimodal understanding capabilities. In contrast, specialized models like R1-Omni and Audio-Reasoner can achieve comparable performance through targeted, emotion-specific post-training adaptation techniques. This suggests that various strategies can be effective in bootstrapping emotional intelligence in MLLMs.
Furthermore, a key observation was the positive correlation between the average number of reasoning steps a model takes and its overall performance. This highlights the importance of encouraging deeper reasoning in MLLMs, prompting them to reflect and identify the triggers behind emotional states, which can significantly enhance their emotional intelligence.
The study also identified limitations in current MLLMs, particularly concerning their visual perception and the effective integration of multimodal information. Many models struggled with subtle facial expressions or failed to accurately interpret emotions when critical audio or visual signals were missing, often due to their inability to process all modalities simultaneously. These findings suggest that the future of emotional intelligence in AI lies in developing more powerful “omnimodal” models capable of seamlessly processing audio, visual, and textual information together.
Also Read:
- Evaluating Continuous Learning in Multimodal AI: Introducing MLLM-CTBench
- MMReID-Bench: A New Standard for Multi-modal Person Re-identification
MME-Emotion is poised to serve as a foundational resource for advancing emotional intelligence in MLLMs, fostering future research and enabling significant advancements in downstream applications across various fields, including education, healthcare, and human-robot interaction. For more information, visit the MME-Emotion project page.
