TLDR: VoiceAgentBench is a new benchmark with over 5,500 multilingual spoken queries, including Indian cultural contexts, designed to evaluate Speech Language Models (SpeechLMs) on complex agentic tasks like multi-tool orchestration, multi-turn dialogues, and adversarial robustness. It reveals significant performance gaps between SpeechLMs and ASR-LLM pipelines, particularly in multilingual generalization, cultural grounding, and safety, highlighting critical limitations of current voice assistants for real-world agentic scenarios.
Voice assistants have become an integral part of our daily lives, from setting alarms to answering complex queries. Behind these seemingly simple interactions are sophisticated Large Language Models (LLMs) and, more recently, Speech Language Models (SpeechLMs) that aim to understand and respond to spoken commands directly. While LLMs have shown remarkable capabilities in reasoning, planning, and executing multi-step tasks with external tools, the world of voice-based agents has largely focused on basic functions.
A new research paper, “VOICEAGENTBENCH: AREVOICEASSISTANTS READY FOR AGENTIC TASKS?” by Dhruv Jain, Harshit Shukla, Gautam Rajeev, Ashish Kulkarni, Chandra Khatri, and Shubham Agarwal from Krutrim AI, delves into a critical question: are current voice assistants truly ready for complex, “agentic” tasks? The authors highlight a significant gap in existing speech benchmarks, which primarily evaluate isolated capabilities like transcription or simple question-answering. These benchmarks fail to assess real-world agentic scenarios that demand multilingual and cultural understanding, robust handling of adversarial inputs, complex tool use, and multi-turn interactions.
Introducing VoiceAgentBench: A Comprehensive Evaluation Framework
To address this void, the researchers introduce VoiceAgentBench (VAB), a groundbreaking benchmark designed to rigorously evaluate SpeechLMs in realistic spoken agentic settings. VAB is unique for several reasons:
- It comprises over 5,500 synthetically generated spoken queries.
- A significant portion of these queries and dialogues are grounded in the rich cultural context of India, supporting English, Hindi, and five other Indian languages (Bengali, Marathi, Tamil, Telugu, and Malayalam). This reflects the true linguistic and cultural diversity of real-world interactions.
- To simulate realistic speaker variability, VAB employs a novel diversity sampling algorithm based on speaker embeddings for Text-to-Speech (TTS) voice conversion, ensuring a wide range of accents, speaking styles, and vocal characteristics.
Diverse Agentic Tasks for Rigorous Testing
VAB categorizes agentic tasks into six distinct types to provide a holistic evaluation:
- Single Tool Call: Simple tasks requiring parameter filling for a given tool.
- Single Tool with Retrieval: Models must select the most relevant tool from a list and then fill its parameters.
- Parallel Tool Calls: Involves selecting and invoking multiple independent tools simultaneously.
- Dependent Tool Calls: A novel and challenging setting where tools are chained sequentially, meaning the output of one tool call feeds into subsequent calls. Examples include booking a cab, ordering food, or managing payments.
- Dialog-Based Tool Invocation: Evaluates tool calls based on multi-turn conversational interactions.
- Safety Evaluations: Assesses the model’s ability to identify and refuse adversarial or unsafe requests.
The evaluation framework measures performance across four key metrics: Tool Selection (accuracy in identifying the correct tool), Tool Call Structure (adherence to the expected output format and schema), Parameter Filling (correctness of arguments, semantically judged by an LLM), and Refusal Rate (the ability to decline harmful requests).
Also Read:
- Unpacking the Open ASR Leaderboard: Accuracy and Efficiency in Speech Recognition
- Evaluating Long Audio Understanding in Language Models: Introducing AUDIOMARATHON
Key Findings: Gaps and Opportunities
The experiments conducted with VAB reveal several critical insights:
- SpeechLMs Lag Behind: There’s a significant performance gap between end-to-end SpeechLMs and traditional ASR-LLM pipelines (where speech is first transcribed by an Automatic Speech Recognition model and then processed by an LLM). While KimiAudio 7B showed the strongest performance among SpeechLMs, it still underperformed compared to ASR-LLM setups.
- Challenges in Multilingual Generalization: SpeechLMs showed a notable decline in performance when moving from English to Hindi and further to other Indic languages, suggesting limited exposure to diverse Indic data during training.
- Cultural Robustness Needed: Both SpeechLMs and ASR-LLM pipelines struggled to maintain accuracy when queries required Indian-context grounding, indicating a clear need for improved cultural understanding.
- Complex Tasks Remain Difficult: Sequential and dependent tool calling tasks proved to be the most challenging for all models, highlighting the complexity of orchestrating multi-step, interdependent workflows.
- Safety Concerns: SpeechLMs demonstrated lower refusal rates for harmful requests, especially in Indic languages, compared to ASR-LLM pipelines, underscoring a critical limitation in safety and instruction-following.
The research also found that ASR errors are a major bottleneck for Indic languages in ASR-LLM pipelines, and that providing one-shot examples significantly boosts SpeechLM performance in tool call structure and parameter filling. Furthermore, while explicit refusal prompts improve safety, models still struggle with adversarial hints.
This comprehensive evaluation underscores that while SpeechLMs have made strides, there are substantial gaps in their ability to handle complex, culturally diverse, and safety-critical agentic tasks. The VoiceAgentBench provides a crucial tool for future research to drive the development of more capable, safe, and culturally inclusive speech agents. You can read the full research paper for more details here.
