TLDR: A new method called “Query Rewriting with Multi-LLM Knowledge Fusion” is proposed to create high-quality speech instruction datasets for Large Speech Language Models (LSLMs) at a low cost. It uses multiple AI models to rewrite text instructions to be more suitable for speech synthesis and then validates the synthesized speech for linguistic accuracy, significantly improving data usability and LSLM performance without extensive human annotation.
Large Speech Language Models (LSLMs) are showing immense promise in understanding and responding to spoken commands, but their full potential has been held back by a significant challenge: the scarcity of high-quality training data. Traditional methods of collecting human-annotated speech instruction datasets are incredibly expensive and time-consuming. While using Large Language Models (LLMs) to generate data has been explored, the gap between AI-generated responses and natural human speech often introduces biases and limitations.
A new research paper, titled “Unlocking Speech Instruction Data Potential with Query Rewriting,” addresses this critical issue by proposing an innovative framework that leverages AI to create high-quality speech instruction datasets automatically and at a much lower cost. The core idea is to intelligently rewrite text instructions before they are converted into speech, ensuring the synthesized audio is more natural and accurate for training LSLMs.
Intelligent Query Rewriting for Better Speech Synthesis
The researchers introduce a “query rewriting framework with multi-LLM knowledge fusion.” This means they use several different LLMs, such as Llama-3-8B-Instruct, Phi-3-small-8k-instruct, and Qwen2-7B-Instruct, to rephrase original text instructions. The goal is to transform text that might be difficult for Text-To-Speech (TTS) models (like abbreviations, complex numbers, or technical terms) into a format that TTS models can synthesize more accurately, without changing the original meaning. By using multiple LLMs, the system benefits from diverse perspectives on how to best rewrite a query.
Ensuring Quality with Multi-Agent Validation
After the text is rewritten and synthesized into speech, the system doesn’t just assume it’s perfect. It employs a “multi-agent annotation and validation” method. This involves using several different Automatic Speech Recognition (ASR) models (like whisper-large-v3, canary-1b, and parakeet-tdt-1.1b) to convert the synthesized speech back into text. Then, multiple embedding models compare this recognized text with the original instruction to measure their semantic similarity. This multi-agent approach helps to catch and correct errors that a single ASR or validation model might miss, ensuring that only high-quality, linguistically equivalent speech data is used for training.
Handling Complex Rewrites with Knowledge Fusion
For particularly challenging queries that even the initial rewriting process might struggle with, the framework incorporates a “knowledge fusion” step. This involves fine-tuning a powerful LLM (like Meta-Llama-3-8B-Instruct) on examples of successfully rewritten queries from the different LLMs. By learning from these successes, the model becomes more adept at handling complex scenarios, such as those requiring context-aware understanding or the integration of specialized knowledge.
Also Read:
- INoT: Empowering AI Agents with Internal Reflection for Enhanced Performance and Efficiency
- elsciRL: A New Open-Source Library for Language Integration in Reinforcement Learning
Significant Improvements and Cost-Effectiveness
The experimental results are highly promising. The proposed method significantly increased the usability of synthesized data from 72% to 93%. It also improved the semantic similarity between the synthesized speech and the original text by 5%. Furthermore, the research highlights a crucial finding for LSLM training: aligning models with high-quality human-annotated responses (referred to as “Golden” data) yields significantly better performance than using LLM-generated continuation data, even if the latter is more abundant.
Perhaps one of the most compelling aspects of this research is its cost-effectiveness. The automated data generation process is estimated to be less than one-tenth the cost of creating comparable high-quality datasets through human annotation, making it a highly practical solution for advancing speech AI.
This work paves the way for building more robust and capable Large Speech Language Models by providing a scalable, automated, and cost-efficient method for generating high-quality speech instruction datasets. For more in-depth details, you can read the full research paper here.
