TLDR: A new method, JSA-SPG, enables pronunciation-lexicon free training for phoneme-based crosslingual ASR. It uses a latent variable model (S2P, P2G) and an auxiliary G2P model, jointly trained with the JSA algorithm. With only 10 minutes of phoneme supervision, it achieves 5% error rate reductions over state-of-the-art methods and significantly improves language domain adaptation, making ASR more accessible for low-resource languages.
In the world of automatic speech recognition (ASR), especially when dealing with many different languages, a common challenge arises: the need for pronunciation lexicons. These lexicons are like dictionaries that map words to their phonetic pronunciations, and they are crucial for phoneme-based ASR systems. However, creating and maintaining these lexicons for thousands of languages is a labor-intensive and costly process, particularly for languages with limited resources.
A new research paper introduces an innovative method called JSA-SPG that aims to eliminate this dependency on pronunciation lexicons. The core idea is to treat phonemes – the basic units of sound in a language – as “hidden variables” within a speech recognition model. This allows the system to learn how to convert speech to text without needing explicit, pre-defined phoneme labels for every word.
The JSA-SPG method is built upon three interconnected models: a speech-to-phoneme (S2P) model, a phoneme-to-grapheme (P2G) model, and an auxiliary grapheme-to-phoneme (G2P) model. Think of it as a pipeline: speech goes in, the S2P model tries to figure out the hidden phonemes, then the P2G model converts those phonemes into written text. The G2P model helps in the learning process by suggesting possible phoneme sequences from text.
The magic behind training these three models together is an advanced algorithm called Joint Stochastic Approximation (JSA). This algorithm is particularly effective for models with discrete hidden variables, like phonemes. What’s remarkable is that JSA-SPG can be trained with very little direct phoneme supervision. The researchers found that initializing the S2P model with a pre-trained multilingual model (named Whistle) and providing just 10 minutes of speech data with phoneme labels was enough to achieve impressive results. This semi-supervised approach significantly reduces the need for extensive, manually annotated data.
Experiments were conducted on Polish and Indonesian languages, which were not part of the initial Whistle model training. The results were highly promising. JSA-SPG achieved a 5% reduction in error rates compared to the best existing crosslingual fine-tuning methods, even those using full phoneme supervision. This highlights the efficiency and effectiveness of the pronunciation-lexicon free approach.
Furthermore, the paper explores how JSA-SPG can be used for language domain adaptation. This is where a model trained on one type of speech data (e.g., Wikipedia articles) needs to perform well on another type (e.g., parliamentary speeches). By using the auxiliary G2P model to generate phoneme labels from text-only data, the P2G model can be further adapted. This approach outperformed the standard practice of simply integrating external language models, showing a 9% error rate reduction. This means the system can adapt to new language styles and vocabularies more effectively, even with just text data.
Also Read:
- Exploring Word Stress Representations in Self-Supervised Speech AI
- Boosting Dialect Classification: The Power of Voice Conversion for Low-Resource Languages
In essence, JSA-SPG offers a scalable and cost-effective solution for developing ASR systems for low-resource languages by removing the bottleneck of pronunciation lexicon creation. The researchers have also open-sourced their training code and complete pipeline to encourage further exploration in this field. You can find more details in the full research paper available at this link.
