TLDR: AImoclips is a new benchmark for evaluating how well text-to-music (TTM) AI systems convey emotions to human listeners. Researchers generated over 1,000 music clips using 12 emotion intents and six TTM models (both open-source and commercial). Human participants rated the perceived valence and arousal of these clips. Findings show commercial models tend to produce more pleasant music, open-source models the opposite, and all models struggle with emotional neutrality, especially with low-arousal emotions. The benchmark provides insights for developing more emotionally aligned TTM systems.
Recent advancements in artificial intelligence have brought about remarkable transformations in creative fields, including music generation. Text-to-music (TTM) systems, which allow users to create music from simple text prompts, have become increasingly popular due to their ease of use and potential for expressive output. However, a crucial aspect often overlooked in the evaluation of these systems is their ability to accurately convey intended emotions to human listeners.
A new research paper introduces AImoclips, a comprehensive benchmark designed specifically to address this gap. Developed by researchers from the Korea Advanced Institute of Science and Technology (KAIST) and Seoul National University (SNU), AImoclips provides a standardized way to assess how well TTM systems communicate emotions, covering both widely available open-source models and commercial platforms.
Building the Benchmark: Emotions, Models, and Human Perception
To create AImoclips, the researchers first selected 12 distinct emotion intents. These emotions were carefully chosen to span the four quadrants of the valence-arousal space – a common model for describing emotions where valence refers to pleasantness (positive to negative) and arousal refers to intensity (calm to excited). Examples include ‘happy’, ‘excited’, ‘energetic’ (high valence, high arousal), ‘angry’, ‘anxious’, ‘scared’ (low valence, high arousal), ‘sad’, ‘gloomy’, ‘dull’ (low valence, low arousal), and ‘relaxed’, ‘calm’, ‘tranquil’ (high valence, low arousal).
Next, six state-of-the-art TTM systems were employed to generate music clips for each of these emotion intents. The selection included four open-source models (AudioLDM 2, MusicGen, Mustango, and Stable Audio Open) and two commercial models (Suno v4.5 and Udio v1.5 Allegro). In total, over 1,000 unique 10-second music clips were generated. To ensure that only the music itself influenced emotional perception, all clips were instrumental, meaning they contained no vocals.
The core of the AImoclips benchmark lies in its human evaluation component. A total of 111 participants were asked to rate the perceived valence and arousal of a selection of these music clips on a 9-point Likert scale. This extensive human feedback allowed the researchers to gather rich, continuous emotion annotations, providing a detailed understanding of how listeners interpret the emotional content of AI-generated music.
Key Findings: Biases and Strengths in AI Music Emotion
The analysis of the AImoclips data revealed several significant insights into the current capabilities and limitations of TTM systems:
-
Commercial vs. Open-Source Differences: Commercial systems like Suno and Udio tended to produce music that human listeners perceived as more pleasant than intended. Conversely, open-source systems often generated music perceived as less pleasant than their intended emotional prompt. This difference might be attributed to factors like audio quality or general listener preference for commercial outputs.
-
High-Arousal Emotions Conveyed Better: Across all models, emotions associated with high arousal (such as ‘excited’ or ‘angry’) were more accurately conveyed to listeners. Low-arousal emotions, like ‘calm’ or ‘gloomy’, proved more challenging for the systems to express effectively.
-
Bias Towards Neutrality: A significant finding was that all TTM systems exhibited a tendency to generate music perceived as emotionally more neutral than the original text prompts. This suggests a current limitation in their ability to express subtle or highly polarized emotional states, indicating that the emotional impact of AI-generated music is often less pronounced than the textual intent.
-
Valence vs. Arousal: The study also indicated that models generally capture intended arousal more successfully than intended valence, meaning the intensity of the emotion was often clearer than its pleasantness.
Also Read:
- AudioCodecBench: A New Standard for Evaluating Audio Codecs in Large Language Models
- Detecting AI’s Poetic Voice: A New Benchmark for Modern Chinese Poetry
Implications for the Future of AI Music
The AImoclips benchmark offers valuable insights for the ongoing development of emotionally intelligent TTM systems. By highlighting model-specific biases and areas where current systems struggle, it provides a clear roadmap for future research. Understanding the specific acoustic and musical features that contribute to biased emotional perception will be crucial for improving the alignment between a generative AI’s intent and a listener’s experience.
This benchmark dataset, with its continuous emotion annotations, can serve as a vital resource for training predictive models of human emotion ratings or fine-tuning generative models to achieve enhanced affective controllability. The full research paper can be accessed here: AImoclips: A Benchmark for Evaluating Emotion Conveyance in Text-to-Music Generation.
