TLDR: A new benchmark called LISTEN evaluates large audio language models (LALMs) to determine if they genuinely understand emotion from speech’s acoustic cues or primarily rely on transcribed text. The study found that current LALMs predominantly “transcribe” rather than “listen,” showing a strong bias towards lexical content and limited ability to interpret acoustic cues, especially in conflicting or non-lexical emotional contexts. This suggests a need for LALMs to better integrate acoustic information for more nuanced emotion understanding.
Large audio language models, or LALMs, have shown impressive abilities in understanding spoken input and generating responses. These models are crucial for applications requiring social and emotional intelligence, where understanding not just the words but also the tone, pitch, and rhythm of speech is vital. However, a fundamental question has remained: do these models truly “listen” to the nuances of speech, or do they simply convert speech to text and then rely on the words alone?
A recent research paper, titled “Do Audio LLMs Really LISTEN, or Just Transcribe? Measuring Lexical vs. Acoustic Emotion Cues Reliance,” delves into this very question. Authored by Jingyi Chen, Zhimeng Guo, Jiyun Chun, Pichao Wang, Andrew Perrault, and Micha Elsner, the study introduces a novel benchmark called LISTEN (Lexical vs. Acoustic Speech Test for Emotion in Narratives) to provide a clearer answer. You can read the full paper here.
The Challenge of Emotion in Speech
Understanding emotion from speech is complex. It requires processing both lexical cues (the actual words spoken) and acoustic cues (how those words are spoken, including pitch, intonation, and rhythm). For instance, sarcasm often involves words that convey a positive literal meaning, but the speaker’s tone of voice indicates a negative or mocking emotion. Humans naturally rely on acoustic cues in such situations to grasp the true intent.
Many current LALMs are built by adapting large text-based language models with speech-text data. While this gives them strong linguistic abilities, it also creates a potential bias: they might prefer lexical cues and treat acoustic information as secondary. Existing benchmarks often include emotionally explicit words, allowing models to perform well by simply reading the transcript, thus potentially overestimating their “listening” capabilities.
Introducing the LISTEN Benchmark
To address this, the researchers developed LISTEN, a benchmark specifically designed to separate how models rely on lexical cues versus acoustic sensitivity in emotion understanding. LISTEN features four controlled conditions that manipulate the relationship between lexical and acoustic cues:
-
Neutral-Text: In this condition, the words themselves are emotionally neutral (e.g., “Kids are talking by the door”), but the audio expresses a specific emotion (like anger or happiness). This isolates the model’s ability to understand emotion purely from acoustic cues.
-
Emotion-Matched: Here, both the words and the acoustic cues align to express the same emotion (e.g., “What the hell is this?” spoken with frustration). This tests how models integrate consistent multimodal information.
-
Emotion-Mismatched: This condition presents a conflict, similar to sarcasm, where the lexical content suggests one emotion (e.g., “Yeah, that’s working out great for me” implying happiness) but the acoustic cues convey a different, often opposing, emotion (like anger or ridicule). This is a critical test of true “listening.”
-
Paralinguistic: This condition removes all lexical content, focusing solely on nonverbal vocalizations like laughter, sighs, or gasps to convey emotion. This tests the model’s ability to interpret emotion from sounds alone.
Within the first three conditions, models were evaluated in three modalities: Text-only, Audio-only, and Text+Audio, to understand their reliance on each input type.
Key Findings: Lexical Dominance Prevails
The evaluation of six state-of-the-art LALMs, including models from Gemini, Qwen, and Baichuan, revealed a consistent pattern: current LALMs largely “transcribe” rather than “listen.”
-
Neutral-Text Condition: In the text-only setting, models achieved near-perfect accuracy by simply identifying the neutral lexical content. However, in the audio-only setting, performance dropped significantly (25-35%), indicating difficulty in recognizing emotion from acoustic cues alone. When both text and audio were combined, some models showed modest gains, but open-weight models often performed worse than their audio-only baselines, suggesting that the neutral text still biased their interpretation, even when the audio was emotional.
-
Emotion-Matched Condition: When lexical and acoustic cues aligned, models showed relatively balanced performance across text and audio inputs, with slight improvements when both were combined. This suggests some cue integration, but overall accuracy remained modest, and a bias towards neutral predictions persisted.
-
Emotion-Mismatched Condition: This was a significant challenge. While LALMs could detect that an emotional discrepancy existed (e.g., recognizing sarcasm), they struggled to identify the specific sarcastic emotion. They often collapsed diverse sarcastic emotions (like anger, disgust, or ridicule) into a narrow set of categories, primarily “ridicule” and “frustration.” This indicates a limitation in reasoning about nuanced emotional meaning in conflicting contexts.
-
Paralinguistic Condition: Performance in this condition was just above random guessing. Models showed limited ability to interpret emotion from nonverbal sounds alone, often defaulting to “neutral” predictions due to the absence of lexical content.
Also Read:
- Voice Input Challenges: How Spoken Queries Increase AI Hallucinations
- Unlocking Dynamic Stress Detection from Speech: A Temporal Progression Approach
Conclusion: More Transcribing, Less Listening
The LISTEN benchmark clearly demonstrates that current large audio language models are heavily biased towards lexical information. They tend to over-emphasize text, often defaulting to “neutral” interpretations when lexical cues are neutral or absent, regardless of strong acoustic signals. Even when acoustic and lexical cues align, the benefits of multimodal integration are limited. In complex scenarios like sarcasm, models can detect conflict but fail to interpret the precise emotional intent.
The researchers conclude that LALMs often “transcribe” more than they truly “listen.” Future advancements in these models could benefit significantly from incorporating techniques used in dedicated Speech Emotion Recognition (SER) systems, which explicitly model prosodic, spectral, and temporal features of speech. The goal for next-generation audio language models should be not just to perceive acoustic variation, but to infer its emotional and communicative intent more accurately.
