TLDR: A new research paper evaluates leading multimodal LLMs (Gemini 2.5 Pro/Flash, Qwen2.5-Omni) on core music perception tasks, revealing a significant “perceptual gap.” Models perform exceptionally well with symbolic MIDI data but show substantial accuracy drops when processing raw audio. Reasoning strategies and few-shot prompting offer limited benefits for audio, indicating that current AI systems can reason effectively over musical symbols but lack reliable “listening” capabilities from audio waveforms. The study highlights the need for stronger audio front-ends in AI to achieve genuine musical understanding.
Large Language Models (LLMs) are increasingly claiming to possess a nuanced understanding of music. However, a recent research paper, “Evaluating Multimodal Large Language Models on Core Music Perception Tasks,” challenges this claim by highlighting a significant gap between an AI’s ability to process musical symbols and its capacity to genuinely “listen” to audio. This study, conducted by Brandon J. Carone, Pablo Ripollés, and Iran R. Roman, rigorously benchmarks three state-of-the-art LLMs: Gemini 2.5 Pro, Gemini 2.5 Flash, and Qwen2.5-Omni.
The researchers focused on three fundamental music skills that demand structural understanding rather than mere surface recognition: Syncopation Scoring, Transposition Detection, and Chord Quality Identification. Syncopation scoring involves recognizing rhythmic expectation violations, transposition detection requires identifying melodies regardless of their absolute pitch, and chord quality identification necessitates recognizing interval patterns within chords.
To thoroughly investigate the models’ capabilities, the study explored three key sources of variability: the input modality (audio versus symbolic MIDI data), the exposure to examples (zero-shot versus few-shot prompting), and different reasoning strategies (Standalone, Chain-of-Thought, and LogicLM). A notable aspect of their methodology was the adaptation of LogicLM, a framework that combines LLMs with deterministic symbolic solvers. This approach helps to ensure that correct answers are not merely masking flawed perceptual analysis, making the distinction between perception and reasoning explicit.
The stimuli used in the experiment were original musical recordings created by a human musician, sourced from The MUSE Benchmark. These recordings included drum excerpts for syncopation, guitar or piano melodies for transposition, and piano chords for quality identification.
The results revealed a clear and consistent “perceptual gap.” While the Gemini models achieved near-ceiling performance when given MIDI inputs, their accuracy dropped significantly when processing raw audio. This suggests that current multimodal LLMs excel at reasoning over symbolic representations of music but struggle to reliably extract musical information directly from sound waves. Qwen2.5-Omni generally underperformed compared to the Gemini models, especially under the LogicLM condition.
Interestingly, advanced reasoning strategies like Chain-of-Thought and LogicLM, along with few-shot prompting, offered only minimal gains in accuracy for audio inputs. This was particularly surprising for audio tasks, where LogicLM, despite its effectiveness with MIDI, remained brittle. For instance, in Transposition Detection, Gemini Pro sometimes preserved sequence length without truly grasping intervallic structure, a flaw that LogicLM helped expose by enforcing musical consistency.
Also Read:
- Unlocking Advanced Reasoning in Audio LLMs with Process-Focused Rewards
- Bridging the Modality Gap: New Training Strategies for Balanced AI Reasoning
The paper concludes that while multimodal LLMs can reason effectively over symbolic music data, they have not yet developed a reliable “listening” ability. This limitation is critical because human beings experience music through audio, not through symbolic proxies. The authors emphasize that true “musical understanding” requires models to process audio tracks directly, much like they handle text or video. They advocate for the development of stronger audio front-ends for LLMs and better propagation of uncertainty into downstream solvers to bridge this perceptual gap. For more details, you can read the full research paper here.
