TLDR: The TSPC model introduces a two-stage, phoneme-centric architecture for Vietnamese-English code-switching Automatic Speech Recognition (ASR). By using an extended Vietnamese phoneme set as an intermediate representation, TSPC effectively handles the unique challenges of mixed-language speech, such as tonal differences and phonological overlaps. It significantly outperforms existing ASR baselines, including PhoWhisper-base, achieving a lower word error rate of 20.8% with fewer training resources, demonstrating a robust solution for complex multilingual ASR scenarios.
Automatic Speech Recognition (ASR) systems have become increasingly common, but they face significant hurdles when dealing with code-switching (CS) – instances where speakers seamlessly switch between two or more languages within a single conversation. This challenge is particularly pronounced for language pairs like Vietnamese and English, which have distinct phonological features and overlapping sounds, making accurate recognition difficult.
Researchers have introduced a novel solution called the Two-Stage Phoneme-Centric model, or TSPC, specifically designed to tackle Vietnamese-English code-switching ASR. This innovative architecture adopts a phoneme-centric approach, utilizing an extended set of Vietnamese phonemes as an intermediate representation. This acts as a linguistic bridge, facilitating more effective modeling of mixed-lingual speech.
The TSPC model is built upon two main stages: a Speech-to-Phone (S2P) module and a Phone-to-Text (P2T) module. The S2P module is responsible for converting acoustic input into sequences of phonemes, essentially breaking down spoken words into their fundamental sound units. Following this, the P2T module takes these phoneme sequences and translates them into coherent text. This two-stage process allows the system to better understand and process the subtle phonological shifts inherent in code-switching scenarios, especially crucial for tonal languages like Vietnamese where pitch variations convey different meanings.
One of the key innovations of TSPC lies in its ability to handle the unique characteristics of Vietnamese, a tonal language, and its interaction with non-tonal English. The model addresses the challenge of phonological overlap, where similar sounds in both languages can lead to recognition errors in traditional ASR systems. By converting English pronunciations into Vietnamese syllables and then into precise phonetic representations, TSPC creates a robust method for cross-linguistic phonetic adaptation.
The development of TSPC involved combining independently pre-trained S2P and P2T models, which were then jointly fine-tuned for code-switched scenarios. The S2P model leverages a pre-trained encoder for acoustic feature extraction, while the P2T model, inspired by machine translation, uses a T5 model design to convert phoneme sequences into text. During fine-tuning, the S2P parameters are frozen to ensure consistent phonetic sequence production, and a cross-attention mechanism allows the P2T encoder to integrate linguistic and acoustic features from the S2P model.
Experimental results have demonstrated that TSPC consistently outperforms existing ASR baselines, including models like Whisper-base and PhoWhisper-base, in Vietnamese-English code-switching ASR. It achieved a significantly lower word error rate of 20.8% while requiring reduced training resources. Even for pure Vietnamese speech recognition, TSPC showed strong performance, achieving an average word error rate of 17.93% with considerably less training data compared to some baselines.
This research highlights the effectiveness of a phoneme-centric approach and the importance of converted phonemes in advancing speech recognition performance in complex code-switching and future multilingual systems. The TSPC architecture represents a significant step forward in making ASR systems more capable of handling the diverse and dynamic nature of human language. You can read the full paper here: TSPC: A Two-Stage Phoneme-Centric Architecture for Code-Switching Vietnamese-English Speech Recognition.
Also Read:
- Exploring the Nuances of Voice: Insights from the First Voice Timbre Attribute Detection Challenge
- Falcon3-Audio: Achieving Top-Tier Audio-Language Understanding with Data Efficiency
The researchers also emphasized the ethical considerations, noting that the TSPC architecture was trained using real Vietnamese pronunciation styles of English words and a diverse dataset from different regions, genders, and ages in Vietnam to mitigate biases. They advocate for further expansion of training data in auditory and text modalities to enhance the model’s generalization capabilities across a wider range of code-switched scenarios.
