TLDR: This research systematically evaluates state-of-the-art Multimodal Large Language Models (MLLMs) for understanding human emotions from text, audio, and video. It benchmarks their performance across various datasets, analyzes how model design and data characteristics influence results, and proposes a new hybrid strategy combining generative knowledge prompting with fine-tuning to significantly boost MLLM capabilities in affective computing.
Understanding human emotions is a complex task, as our expressions are inherently multimodal, involving not just words but also facial expressions, body language, and tone of voice. This field, known as Multimodal Affective Computing (MAC), aims to interpret these emotions by integrating information from various sources like text, video, and audio.
Traditionally, MAC approaches often relied on pre-processed features, which limited their ability to deeply explore and learn emotional information. However, the emergence of Multimodal Large Language Models (MLLMs) has brought a significant shift. These advanced models, like GPT-4V, LLaVA, and Gemini, combine the powerful language understanding and reasoning abilities of Large Language Models (LLMs) with the capacity to process and align information from diverse modalities, offering a unified framework for affective computing.
Despite their immense potential, MLLMs face challenges in practical MAC applications, including varying performance across complex tasks and a lack of clear understanding of how their design and data characteristics influence emotional analysis. To address these issues, a recent study conducted a systematic benchmark evaluation of several state-of-the-art open-source MLLMs. These models, including HumanOmni, Qwen2.5Omni, VideoLLaMA2-AV, Ola, MiniCPM-o 2.6, Emotion-LLaMA, and PandaGPT, were tested on multiple established MAC datasets such as CMU-MOSI, CMU-MOSEI, CH-SIMS, CH-SIMS v2, MELD, and UR-FUNNY v2.
The evaluation not only compared the performance of these MLLMs against each other and traditional machine learning methods but also provided valuable insights into model optimization. The researchers analyzed how different model architectures—such as modality alignment mechanisms, fusion strategies, and model size—and dataset properties, like modality dominance, impact performance in affective analysis tasks.
Furthermore, the study introduced a novel hybrid strategy to enhance MLLMs’ affective computing capabilities. This approach combines generative knowledge prompting with supervised fine-tuning. It works by first using the MLLMs’ zero-shot ability to extract descriptions from raw audio and video inputs. These extracted cues are then incorporated into knowledge-guided prompts, followed by supervised fine-tuning on this augmented input. Experimental results showed that this integrated method significantly improved performance across various MAC tasks, outperforming standalone fine-tuning methods.
For instance, on the CMU-MOSI dataset, MLLMs generally performed exceptionally well, largely due to the text modality’s dominant role, allowing MLLMs to leverage their strong language understanding. However, on CMU-MOSEI, performance varied, partly due to dataset imbalances. In datasets like CH-SIMS and CH-SIMS v2, where modality contributions are more balanced, MLLMs demonstrated stronger advantages, highlighting their ability to fuse and process multimodal information effectively.
The research also delved into the impact of individual modalities. On datasets like CH-SIMS and CMU-MOSI, the text modality consistently showed an advantage across all MLLMs. Models like HumanOmni, Qwen2.5Omni, and MiniCPM-o performed particularly well in audio processing, often because their audio encoders (like Whisper) were pre-trained on speech-to-text tasks, making them highly effective for standalone audio analysis. Qwen2.5Omni also showed a significant advantage in visual modality processing, indicating its robust visual feature extraction and fusion mechanisms.
Also Read:
- Advancing Medical AI: A Deep Dive into Reasoning Capabilities of Large Language Models
- The AI Evolution in Virtual Worlds: How Language Models are Redefining VR Gaming
This systematic evaluation and the proposed hybrid strategy offer promising avenues for future research and development in Multimodal Affective Computing. The code for this research is openly available, encouraging further exploration and optimization of MLLMs in more complex and diverse MAC scenarios. You can find more details in the full research paper available here.
