TLDR: SAGE-LD is a novel end-to-end neural model for language diarization that can identify language shifts in audio, even in code-switching scenarios. It features a learnable query-based architecture, multilingual awareness, and is extensively pretrained on simulated code-switching data. This approach allows SAGE-LD to support an unconstrained number of languages and generalize effectively across diverse real-world environments. The model achieves state-of-the-art performance on various benchmarks, demonstrating significant improvements over existing methods by addressing data scarcity and architectural limitations.
Language diarization (LD) is a crucial task in speech processing that involves identifying which language is spoken at a specific moment within an audio stream. This capability is particularly vital in scenarios involving code-switching, where a single speaker might alternate between multiple languages within or across their utterances. Such linguistic behavior presents significant challenges for traditional multilingual speech processing systems due to the varying phonetic, syntactic, and lexical properties across languages.
Accurate language diarization allows for the breakdown of code-switched utterances into their individual monolingual segments. This segmentation is important because it enables the application of language-specific downstream systems, which often perform better under constrained conditions compared to more general multilingual models.
Historically, research in language diarization has followed two main paths. One approach integrates LD as a part of code-switching automatic speech recognition (CS-ASR), where language boundaries are either implicitly modeled or explicitly marked to aid multilingual transcription. While these methods can achieve high accuracy, they typically assume a single language pair, limiting their broader applicability. The second approach treats LD as a standalone task, similar to speaker diarization, which aims to segment speech by speaker identity. These systems often use multi-stage pipelines or end-to-end neural diarization methods to process multiple languages.
Recent efforts have focused on developing more general-purpose LD models that can handle multiple languages within a single framework. However, performance in these setups still lags behind speaker diarization, and coverage has often been restricted to a fixed set of languages.
Introducing SAGE-LD: A Scalable and Generalizable Solution
To overcome these limitations, researchers have introduced SAGE-LD, a comprehensive framework designed for end-to-end language diarization that supports an unconstrained number of languages. SAGE-LD integrates a learnable query-based architecture with multilingual awareness and large-scale pretraining on simulated code-switching data. This combination helps address the challenges of data scarcity and architectural optimization, allowing the model to generalize effectively to real-world multilingual environments.
The SAGE-LD model processes raw audio waveforms and generates diarization outputs through three main components:
-
Multilingual Feature Extractor: This component uses the convolutional layers from multilingual self-supervised speech models (S3Ms) to extract language-agnostic acoustic features. By omitting the Transformer layers, it avoids mixing linguistic information, ensuring that the extracted features capture universal speech characteristics.
-
Contextual Encoder: Conformer layers are stacked to capture language-aware context. Unlike previous models that might aggregate features using sliding windows, SAGE-LD’s design directly models frame interactions, preserving fine-grained temporal cues crucial for diarization performance.
-
Masked Attention Decoder: Inspired by segmentation models, this decoder uses learnable language queries. It employs a small number of queries, with one specifically reserved for voice activity detection (VAD), and iteratively refines predictions to identify language shifts.
Innovative Data Augmentation and Training
A key innovation in SAGE-LD is its approach to data augmentation. To address the scarcity of multilingual LD data, the model is pretrained on a large-scale simulated corpus of code-switching utterances. A critical challenge in creating simulated data is disentangling speaker shifts from language shifts. SAGE-LD tackles this by using UniCoM, which incorporates a voice conversion model to unify speaker identity across utterances, ensuring that the model learns to identify language changes rather than speaker changes.
The training strategy is two-staged: first, language family-based pretraining on the simulated corpus, which equips the model with general diarization capabilities and transfers language knowledge. Second, the model undergoes dataset-specific adaptation using a small amount of real-world annotated data to capture domain-specific characteristics.
Also Read:
- Emo-TTA: Enhancing Speech Emotion Recognition in Dynamic Environments
- Shedding Light on Speech-to-Text Decisions with Contrastive Explanations
State-of-the-Art Performance
Experimental results demonstrate that SAGE-LD achieves state-of-the-art performance across several language diarization benchmarks. It consistently shows superior results in both long-form conversational and short-form broadcast settings, with relative performance improvements ranging from 23% to 52% over previous methods. This robust performance is particularly impressive given that SAGE-LD uses a smaller number of parameters compared to many prior models.
Ablation studies further confirmed the effectiveness of SAGE-LD’s design choices. For instance, applying additional attentive pooling over acoustic features was found to degrade performance, supporting the model’s approach of avoiding such aggregation. The specialized loss functions, including focal loss and focal Tversky loss, also contributed significantly to the model’s accuracy, especially in handling the sparse occurrence of embedded languages.
SAGE-LD represents a significant advancement in language diarization, offering a scalable and generalizable framework capable of handling an unbounded number of languages. This work not only pushes the boundaries of language diarization research but also lays a foundational framework for broader developments in code-switching speech technologies. For more details, you can read the full research paper here.
