TLDR: A new method called Serialized Output Prompting (SOP) significantly improves Large Language Model (LLM)-based multi-talker automatic speech recognition (MT-ASR). SOP works by extracting preliminary, speaker-specific transcriptions from mixed speech using a Separator and CTC layers, then using these “serialized output prompts” to explicitly guide the LLM decoder. A three-stage training strategy enables effective learning, leading to substantial performance gains on the LibriMix dataset, especially in complex three-talker scenarios, by helping LLMs better handle overlapping speech.
Imagine trying to understand multiple people speaking at once. That’s the challenge Multi-Talker Automatic Speech Recognition (MT-ASR) systems face. While standard ASR focuses on a single speaker, MT-ASR aims to transcribe all speakers’ content from overlapping speech, a significantly more complex task. Recent advancements have seen Large Language Models (LLMs) integrated into MT-ASR, showing promising improvements. However, these LLM-based systems often use very basic prompts or no prompts at all, missing an opportunity to fully leverage the LLM’s capabilities, especially in highly complex scenarios like three or more speakers talking simultaneously.
Researchers at SB Intuitions have introduced a novel approach called Serialized Output Prompting (SOP) for LLM-based MT-ASR. This method explicitly guides the LLM using structured prompts to significantly enhance performance, particularly in challenging overlapping speech conditions. The core idea behind SOP is to provide the LLM with a clear, serialized representation of who said what and when, even before the LLM performs its main transcription task.
To achieve this, the system incorporates two key components after the speech encoder: a Separator and specialized Connectionist Temporal Classification (CTC) layers. The Separator works to disentangle the mixed speech into individual talker-specific content. Following this, the CTC layers, one for each speaker, decode these separated speech segments into token sequences. These sequences are ordered based on when each speaker began talking, creating a “serialized output.” This serialized output, essentially a preliminary transcription of who spoke first, second, and so on, then becomes the Serialized Output Prompt (SOP) for the LLM decoder.
The training of this sophisticated system is broken down into a three-stage strategy to ensure effective and stable optimization. The first stage involves fine-tuning the model with Serialized Output Training (SOT). Here, the speech encoder learns to process mixed speech, and the LLM decoder learns to serialize mixed token sequences. In the second stage, the focus shifts to extracting serialized speech information. The Separator and CTC modules are trained to disentangle speaker-aware features and align them. Finally, the third stage is dedicated to SOP-based adaptation. During this phase, an additional set of LoRA (Low-Rank Adaptation) modules are introduced into the LLM, specifically to adapt it to the SOPs generated by the Separator and CTC layers. Only these LoRA parameters are trained in this stage, allowing the LLM to effectively utilize the guiding prompts.
Experiments conducted on the LibriMix dataset, which includes both two-talker and three-talker scenarios with noisy and clean speech, demonstrated the significant impact of the proposed SOP approach. While the baseline LLM-based SOT model performed well in two-talker conditions, it struggled with the more complex three-talker scenarios. The SOP-based MT-ASR, however, showed substantial performance improvements under both two- and three-talker conditions. This highlights that explicitly providing guiding cues through SOPs is highly effective in improving the LLM’s decoding performance in multi-talker environments. Interestingly, the research also found that simply increasing the LLM’s parameter size (e.g., from 3B to 8B) did not automatically lead to better performance, suggesting that the amount of training data plays a crucial role in fully leveraging larger models.
Also Read:
- Leveraging TV Subtitles to Improve Speech Recognition Accuracy
- Enhanced Silent Speech Recognition: A Two-Stage AI Breakthrough
The analysis of the serialized CTC layers revealed that even in highly overlapping regions or where speakers frequently switched, the system was able to produce good quality, well-aligned speech content for different talkers. This ability to extract accurate preliminary information is what makes SOP a powerful tool for guiding LLMs in complex MT-ASR tasks. This research paves the way for more robust and accurate speech recognition systems capable of handling the intricacies of real-world multi-speaker conversations. For more technical details, you can refer to the full research paper: Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognition.
