TLDR: The MUSE Benchmark, a new open-source resource with 10 music perception tasks, evaluates leading audio LLMs (Gemini Pro/Flash, Qwen2.5-Omni, Audio-Flamingo 3) against human performance. The study reveals a significant gap between AI and humans, particularly in abstract and music-theoretic reasoning, with many models performing at or near chance. Chain-of-Thought prompting proved inconsistent and often harmful, and increasing in-context examples did not reliably improve model learning. The research concludes that current audio LLMs lack the invariant musical representations for deep understanding, necessitating fundamental changes in AI training paradigms.
Recent advancements in Artificial Intelligence have brought forth Multimodal Large Language Models (MLLMs) that show promise in understanding audio. However, a new research paper introduces a critical evaluation tool, the Music Understanding and Structural Evaluation (MUSE) Benchmark, which suggests that current evaluations might be overlooking fundamental weaknesses in how these AI models perform relational reasoning in music.
Authored by Brandon James Carone, Iran R. Roman, and Pablo Ripoll´es, the paper, titled “THE MUSE BENCHMARK: PROBING MUSIC PERCEPTION AND AUDITORY RELATIONAL REASONING IN AUDIO LLMS,” highlights a significant gap between state-of-the-art AI models and human perception when it comes to understanding musical structure and relationships. You can read the full research paper here: The MUSE Benchmark Paper.
Introducing the MUSE Benchmark
The MUSE Benchmark is an open-source resource featuring 10 distinct tasks designed to thoroughly test fundamental music perception skills. These tasks are divided into two tiers: “Beginner” and “Advanced,” with their design rooted in established music cognition research.
The five Beginner tasks focus on core perception and auditory invariances, skills that are robust even in non-musicians. These include:
- Instrument Identification: Classifying instruments based on their unique sound qualities.
- Melody Shape Identification: Recognizing the overall contour of a melody (e.g., ascending, descending).
- Oddball Detection: Identifying out-of-key notes within a harmonic context.
- Rhythm Matching: Determining if two rhythmic sequences are identical.
- Pitch Shift Detection: Recognizing a melody even when its overall pitch has been changed (transposed).
The five Advanced tasks target skills that typically require formal musical training, demanding explicit knowledge of music theory and the ability to track functional relationships over time. These include:
- Chord Identification: Distinguishing between major and minor chords.
- Syncopation Comparison: Identifying which of two rhythms is more syncopated.
- Key Modulation Detection: Recognizing if a change of musical key occurs within an excerpt.
- Chord Sequence Matching: Determining if two sequences of chords are the same.
- Meter Identification: Inferring the underlying beat grouping (e.g., groups of 3 or 4) from a rhythm.
Evaluating AI Against Humans
The researchers evaluated four leading MLLMs: Gemini Pro, Gemini Flash, Qwen2.5-Omni, and Audio-Flamingo 3. These models were tested under two prompting conditions: “Standalone” (mirroring human instructions) and “Chain-of-Thought” (CoT), where models were guided through a multi-step analytical process. A large human baseline of 200 participants, including a subgroup of expert musicians, was also established for comparison.
Key Findings: A Persistent Gap
The results revealed a wide variance in the capabilities of the state-of-the-art models and a consistent gap when compared to human experts. While Gemini Pro showed strong performance on basic perception tasks, other models like Qwen and Audio Flamingo 3 often performed at or near chance levels, indicating severe perceptual deficits.
Human listeners, particularly expert musicians, consistently outperformed most models on tasks requiring abstract reasoning, such as Melody Shape Identification and Pitch Shift Detection, and especially on all tasks in the Advanced tier that demand music-theoretic knowledge. For example, expert musicians achieved perfect accuracy (100%) on Pitch Shift Detection, while the best model, Gemini Pro, only reached 81.36% in the Standalone condition.
Interestingly, the study found that Chain-of-Thought (CoT) prompting, often used to improve AI reasoning, provided inconsistent and frequently detrimental results. In many cases, CoT either had a negligible effect or actively worsened performance, suggesting that step-by-step textual reasoning isn’t a reliable way to enhance models’ non-linguistic perceptual skills.
Furthermore, the research highlighted a fundamental divergence in learning patterns. While human musical training consistently improved performance across tasks, providing more in-context examples (shots) to the models did not reliably enhance their accuracy on complex musical tasks. This suggests that models’ performance is more dependent on their pre-trained capabilities rather than their ability to learn abstract rules from additional examples during inference.
Also Read:
- Unlocking Adaptability: New Benchmark for Editing Auditory Knowledge in AI Models
- MuseTok: A New Approach to Understanding and Generating Symbolic Music with AI
Conclusion: A Call for Deeper Understanding
The MUSE Benchmark serves as a critical diagnostic tool, revealing that current audio LLMs often lack the invariant representations necessary for deep musical reasoning. The findings challenge the AI community to move beyond surface-level classification and to develop foundation models that target genuine perceptual competence. Bridging the human-machine gap in music understanding will likely require fundamental changes in model architecture and training paradigms, rather than simply scaling existing methods with more data or more complex prompts.
