TLDR: This research introduces a novel “thinking while listening” paradigm for speech large language models (LLMs) to improve their reasoning abilities and reduce response latency. By fine-tuning multi-stream LLMs with Chain-of-Thought (CoT) and using a “Question Completeness” metric, models can start reasoning before a user’s query ends. Further preference tuning with Direct Preference Optimization (DPO) significantly reduces latency by 70% without sacrificing accuracy, making speech LLMs more responsive and intelligent.
Voice-based artificial intelligence has become increasingly common, enabling seamless spoken interactions with our devices. However, these systems, often powered by large language models (LLMs), still face significant hurdles when it comes to complex reasoning tasks. While text-based LLMs have seen remarkable improvements in reasoning through techniques like Chain-of-Thought (CoT) prompting, applying these advancements to speech LLMs presents unique challenges, particularly concerning response latency.
A recent research paper titled “Can Speech LLMs Think while Listening?” explores how to enhance the reasoning capabilities of speech LLMs while simultaneously reducing the delay in their responses. The authors, including Yi-Jen Shih from The University of Texas at Austin and researchers from Meta Superintelligence Labs, introduce novel methods inspired by human cognitive processes to achieve this balance.
The Challenge of Reasoning and Latency in Speech LLMs
Traditionally, voice agents operate in a cascaded manner: automatic speech recognition (ASR) converts speech to text, a text-based LLM processes it, and text-to-speech (TTS) converts the response back to audio. Speech LLMs offer a more integrated approach, directly handling speech input and output, which inherently reduces some latency. Yet, they often lag behind their text counterparts in tasks requiring deep reasoning.
Chain-of-Thought (CoT) involves a model generating intermediate reasoning steps before providing a final answer. While effective for accuracy, longer CoT sequences can increase decoding latency, which is a critical factor for natural spoken interactions. Previous attempts to overlap CoT with speech, known as “thinking while speaking,” had limitations like optimal chunk size tuning and potentially vocalizing too much reasoning.
Introducing “Thinking while Listening”
The researchers propose a new paradigm: “thinking while listening.” This approach allows the speech LLM to begin its reasoning process even before the user’s query is fully articulated, much like how humans often start formulating a response or understanding a question before it’s completely finished. This is achieved by leveraging multi-stream speech LLM architectures, which can concurrently process user audio and internal reasoning tokens.
The study builds upon the Moshi model, a full-duplex multi-stream architecture, fine-tuning it to generate CoT within its text monologue stream. A crucial finding was that text-based CoT is as effective as speech-based CoT for improving reasoning in speech LLMs, while being twice as token-efficient.
Key Innovations for Early Reasoning
To enable this “thinking while listening” capability, two primary methods were introduced:
1. Question Completeness (QC) Metric: This novel metric helps the model identify the “inflection point” in a user’s question – the moment when enough information is available to start reasoning. Instead of relying on fixed time shifts, QC uses Kullback-Leibler (KL) divergence to semantically assess how complete a question is. This allows the model to initiate its CoT at the most appropriate time, even if the user is still speaking.
2. Preference Tuning with Direct Preference Optimization (DPO): The researchers observed that standard fine-tuning sometimes struggled with adaptive reasoning or generated overly long CoT traces. To address this, they created preference data using rejection sampling, focusing on correctness and reasoning length. By applying DPO, the model learned to adapt its reasoning dynamically to new input and generate more concise, yet accurate, CoT traces.
Additionally, the model was trained to interleave CoT tokens with streaming ASR (Automatic Speech Recognition) tokens on the text monologue stream. Special switching tokens (`
Also Read:
- AI Agents Learn to Think on the Fly in Full-Duplex Dialogues
- MixReasoning: A Smart Approach to Efficient Language Model Thinking
Impressive Results and Future Implications
The researchers curated a suite of spoken reasoning tasks from well-known text benchmarks (like ARC, PIQA, SIQA, and GSM8K) to evaluate their methods. The results were significant:
- CoT fine-tuning led to an average 2.4x improvement in accuracy over the Moshi baseline across spoken reasoning tasks.
- The “thinking while listening” approach, guided by the Question Completeness metric, offered superior control over the accuracy-latency trade-off compared to heuristic-based methods.
- Further preference tuning with DPO resulted in a remarkable 70% reduction in latency without any loss in accuracy.
This work marks a significant step forward for speech LLMs, enabling them to process and reason concurrently with incoming audio. This capability brings these AI systems closer to the natural, responsive flow of human conversation, paving the way for more intuitive and efficient voice-based interactions. For more details, you can refer to the full research paper here.
