TLDR: This research introduces the Emotional Rationale Verifier (ERV), a new method to help Multimodal Large Language Models (MLLMs) generate explanations for emotions that are consistent with their predictions. Current MLLMs often produce contradictory explanations, eroding user trust. ERV, combined with an Explanation Reward, guides the model to create more faithful and emotionally aligned reasoning during training, without needing extra human-annotated data. The approach significantly improves explanation-prediction consistency and emotional accuracy, as validated by new metrics and human evaluations, paving the way for more trustworthy human-computer interactions.
The way we interact with computers is constantly evolving, moving beyond simple commands to more natural and emotionally intelligent conversations. At the heart of this shift are Multimodal Large Language Models (MLLMs), which can understand information from various sources like video and audio. A crucial aspect of this advanced interaction is the ability of these systems to understand human emotions and, just as importantly, to explain why they’ve identified a particular emotion. This transparency is vital for building user trust.
However, current MLLM-based systems often fall short in this area. They might predict an emotion correctly but then provide an explanation that doesn’t quite match, or even contradicts, their own prediction. This inconsistency can lead to misunderstandings and erode confidence in AI systems, especially in sensitive applications like psychological counseling or educational guidance.
To tackle this challenge, researchers have introduced a novel approach called the Emotional Rationale Verifier (ERV) and an associated Explanation Reward. This method aims to guide MLLMs to produce reasoning that is explicitly consistent with the target emotion during multimodal emotion recognition. What’s particularly innovative is that it achieves this without needing to alter the model’s core architecture or requiring additional, costly human-annotated video-description pairs.
How the Emotional Rationale Verifier Works
The core idea behind ERV is to leverage the inherent linguistic and emotional knowledge within large language models to evaluate the consistency between a given emotion and a generated explanation. Since directly incorporating a large, complex LLM for every evaluation can be computationally expensive, the researchers distilled this knowledge into a more lightweight model: the Emotional Rationale Verifier (ERV).
This ERV module is integrated into a reward-driven training process. During this process, the MLLM receives an ‘Explanation Reward’ based on how well its generated explanation aligns with the true emotion. To make this reward even more precise, the system breaks down each explanation into individual sentences. It then filters out sentences that are emotionally neutral (for example, those describing background details or appearance) and assigns rewards based on the proportion of emotionally relevant sentences that accurately reflect the target emotion.
The ERV itself is trained on a specially constructed dataset. Recognizing the limitations of existing datasets in terms of size and emotional diversity, the researchers created a pseudo-labeled dataset of 20,000 emotional text-label pairs. They achieved this by generating textual explanations from existing video datasets (DFEW and MAFW) and then using a powerful closed-source LLM (GPT-4.1) to assign emotion labels to these descriptions. To further balance the dataset, they augmented underrepresented emotion categories by generating additional descriptions.
New Ways to Measure Emotional Coherence
To thoroughly evaluate the emotional consistency of the generated explanations, the research introduces three new metrics:
- **Explanation Emotion Accuracy (EEA):** Measures how well the explanation aligns with the target emotion.
- **Explanation–Prediction Consistency (EPC):** Quantifies the internal consistency between the explanation and the emotion predicted by the model.
- **Faithful Consistency Rate (FCR):** A strict metric that requires all three components—explanation, predicted emotion, and target emotion—to be in agreement.
Also Read:
- Gaze-VLM: Enhancing AI’s Understanding of Human Actions Through Eye Gaze
- Bridging the Gap: How Symbolic AI Enhances Transparency and Reasoning in Large Language Models
Promising Results
Extensive experiments on the MAFW and DFEW datasets demonstrate the effectiveness of this new approach. The model not only maintains its performance in emotion classification but also significantly improves the consistency between generated explanations and predicted emotions, as measured by the new EEA, EPC, and FCR metrics. For instance, the Faithful Consistency Rate (FCR) saw substantial increases, indicating a much better alignment across explanations, predictions, and ground-truth emotions.
Human evaluations further supported these findings, confirming that the model produces more coherent and emotionally aligned reasoning compared to previous methods. This leads to more reliable and interpretable interactions in various human-computer applications. The robustness of these evaluation metrics was also validated by testing them with different closed-source LLMs, showing consistent improvements.
In conclusion, this research marks a significant step towards creating truly emotionally intelligent human-computer interaction systems by ensuring that MLLMs can provide not just accurate emotion recognition, but also trustworthy and emotionally coherent explanations. You can read the full paper here: Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier.
