TLDR: A new research paper introduces a style-controllable speech augmentation model to improve speaker diarization systems. The model generates diverse speech styles while preserving speaker identity, addressing the issue of intra-speaker variability (e.g., changes in emotion or speaking speed). By blending speaker embeddings from original and augmented audio, the system significantly reduces diarization error rates, achieving 49% and 35% reductions on emotional speech and truncated AMI datasets, respectively, by better grouping varied speech segments from the same individual.
Speaker diarization, the technology that answers “who speaks when” in an audio recording, is vital for applications like live captioning and multi-speaker automatic speech recognition. While significant progress has been made, these systems often struggle with a common challenge: intra-speaker variability. This refers to changes in a single person’s voice due to factors like emotion, health, or speaking speed. When someone raises their voice in anger or speaks faster, a diarization system might mistakenly classify these different vocal styles as belonging to different individuals, leading to errors.
Researchers at Yonsei University and Qualcomm Technologies have proposed a novel approach to tackle this problem using style-controllable speech augmentation. Their work, detailed in the paper “Mitigating Intra-Speaker Variability in Diarization with Style-Controllable Speech Augmentation”, introduces a system that generates diverse speech styles while preserving the speaker’s identity. This augmented speech is then used to make diarization systems more robust.
The core idea is to enrich the training data with various speaking styles from the same speaker. Since large datasets covering diverse styles for each speaker are scarce, the team developed a sophisticated speech generation model. This model integrates concepts from Global Style Tokens (GST) and Vevo, allowing it to learn style representations without explicit labels and offer flexible control over stylistic variations like emotion and expressiveness during generation.
The proposed framework operates in three main stages. First, an initial diarization system processes an audio recording to identify preliminary speaker segments. These segments, which are specific to individual speakers, are then fed into the style-controllable data augmentation model. This model generates new speech samples that maintain the original speaker’s identity but exhibit a wide range of phonetic and stylistic diversity.
Finally, speaker embeddings—numerical representations of a speaker’s voice—are extracted from both the original and the newly generated augmented audio. These embeddings are then blended and re-clustered. By combining the original and augmented data, the system gains a more comprehensive understanding of a speaker’s vocal characteristics across different styles, making it much better at grouping segments from the same speaker, even when their voice changes.
The effectiveness of this approach was validated on two datasets: a simulated emotional speech corpus and a truncated version of the AMI corpus, which contains real conversational data. The results were impressive, showing significant reductions in error rates. On the emotional speech dataset, the system achieved a 49% decrease in error rate, primarily by reducing confusion where one speaker’s varied styles were mistaken for multiple speakers. On the truncated AMI dataset, particularly for shorter segments (15-30 seconds) where diarization is typically more challenging due to limited data, the system saw error rate reductions of 22% and 35% respectively. This demonstrates that augmentation effectively compensates for the scarcity of speaker embeddings in shorter recordings.
Visualizations of speaker embeddings further illustrated the impact. Before augmentation, a single speaker’s voice might be split into multiple clusters due to stylistic variations. After incorporating augmented samples, these disparate clusters merge into coherent groups, effectively ‘bridging’ the initially separated styles and improving same-speaker grouping.
Also Read:
- Enhancing Speech LLMs: A Dual-Channel Approach to Overcome Forgetting and Modality Gaps
- Challenging Audio Security: Discrete Optimal Transport as a Black-Box Attack
This research offers a promising solution to a long-standing challenge in speaker diarization. By leveraging advanced speech generation techniques, the system can create a richer, more diverse representation of individual speakers, leading to more accurate and robust diarization, especially in real-world scenarios where intrinsic intra-speaker variability is common.
