TLDR: Multi-Bench is the first benchmark designed to evaluate the emotional intelligence of Spoken Dialogue Models (SDMs) in multi-turn interactive dialogues. It features a hierarchical structure with basic and advanced tasks, covering emotion understanding, reasoning, and application. The benchmark uses a reproducible evaluation framework and reveals that while current SDMs perform well on basic tasks, they still struggle with advanced multi-turn emotional interactions, especially in English. GPT-4o generally performs best, followed by Step Audio 2, highlighting the ongoing challenges in developing emotionally intelligent conversational AI.
Spoken Dialogue Models (SDMs) have seen rapid advancements, but their ability to handle complex, multi-turn conversations with emotional intelligence has largely remained unexplored. Most existing evaluation methods focus on simple, single-turn interactions, which doesn’t fully capture the nuances of real-world dialogue.
To address this critical gap, researchers have introduced Multi-Bench, a groundbreaking benchmark specifically designed to assess the emotional intelligence of SDMs in genuinely interactive, multi-turn conversations. This new benchmark moves beyond basic speech recognition to evaluate how well these models can understand, reason about, and apply emotions over extended dialogues.
Multi-Bench features a clever hierarchical structure to provide a comprehensive assessment. It includes a basic track that focuses on fundamental emotion understanding and reasoning, and an advanced track dedicated to emotion support and application. This allows for a detailed evaluation of an SDM’s capabilities, from recognizing simple emotions to engaging in complex, emotionally aware interactions.
The benchmark comprises five carefully designed tasks and approximately 3,200 samples. These tasks range from straightforward emotion recognition to more intricate reasoning and interactive dialogue scenarios. To ensure a robust evaluation, Multi-Bench is supported by a reproducible framework that considers both linguistic and acoustic aspects of an SDM’s responses, at both the utterance and conversation levels.
The evaluation framework simulates real-world interactions by constructing user profiles with specific scenarios and goals. User responses are generated by a chat Large Language Model (LLM) and converted into emotional speech, which then serves as input to the SDM. The SDM, in turn, generates both spoken and textual outputs, creating an end-to-end audio-based conversational exchange. This loop continues until a natural termination condition is met, such as the user feeling sufficient emotional relief.
To ensure the emotional expressiveness of the simulated user, an emotion conditioning mechanism is employed. Another LLM determines the most suitable emotion for a given context, and a matching audio prompt from a curated emotional speech dataset is used to generate human-like emotional speech via a Text-to-Speech (TTS) module.
The data for Multi-Bench is curated from various open-source datasets, including UnderEmotion, NVSpeech, PsyQA, PsyDTCorpus, and MultiDialog. These sources cover a diverse range of topics, from everyday conversations to psychological support, ensuring a broad and realistic evaluation scope.
Experiments were conducted on six representative SDMs, including prominent models like GPT-4o, Qwen 2.5 Omni, GLM 4 Voice, Step-Audio-AQAA, Step Audio 2, and Kimi Audio. The results showed that while current SDMs perform well on basic emotion understanding tasks, there is significant room for improvement in advanced multi-turn interactive dialogue and reasoning-related tasks, particularly concerning emotion awareness and application.
GPT-4o generally demonstrated the best overall performance across the benchmark, closely followed by Step Audio 2. Interestingly, the performance gaps between SDMs were minor in daily conversations but became much more pronounced in emotion-centric dialogues, highlighting the greater challenge emotional intelligence poses for conversational AI. For more technical details, you can refer to the original research paper here.
Also Read:
- Speech-DRAME: A New Standard for Evaluating AI Speech Role-Play
- Bridging Audio and Language for Southeast Asia: Introducing SeaLLMs-Audio
Multi-Bench is expected to serve as a vital resource, driving future research and development in the field of emotionally intelligent Spoken Dialogue Models, pushing the boundaries of how AI can interact with humans in a more empathetic and understanding way.
