TLDR: This paper introduces methods to prevent “stability hallucinations” (like repetitive or omitted speech) in LLM-based Text-to-Speech (TTS) models. It proposes the Optimal Alignment Score (OAS) to measure text-speech alignment quality and integrates it into model training. Additionally, it uses attention values from pre-trained models to guide student models via a chain-of-thought approach, significantly reducing errors in synthesized speech without compromising naturalness. The techniques eliminate the need for explicit forced alignment labels.
Large Language Model (LLM)-based Text-to-Speech (TTS) systems have made significant strides in generating natural and expressive human-like speech. These advanced models are particularly good at handling temporal sequences and are versatile for multi-modal tasks. However, despite their capabilities, they often struggle with a specific issue known as “stability hallucinations,” especially when processing long or complex texts. This can lead to undesirable outcomes like repetitive speech, endless audio generation, or even the omission of parts of the intended speech.
Traditional TTS models often relied on explicit text-speech alignment mechanisms, which helped ensure that each part of the text corresponded correctly to a segment of speech. LLM-based TTS models, on the other hand, typically use a “decoder-only” architecture and generate speech by predicting the next token, without a clear, built-in alignment process. While large datasets help them perform well in common scenarios, this lack of explicit alignment becomes a problem with more challenging inputs.
Introducing the Optimal Alignment Score (OAS)
To tackle these stability issues, a new research paper proposes an innovative approach that improves and leverages the attention mechanism within LLM-based TTS models. The first key contribution is the introduction of a metric called the Optimal Alignment Score (OAS). This score is designed to evaluate the quality of alignment between text tokens (the individual units of text) and speech tokens (the individual units of speech) within the LLM. It uses a technique called the Viterbi algorithm to find the best possible alignment path, ensuring continuity and monotonicity – meaning the speech should progress smoothly and in order with the text.
The researchers found a strong correlation between a low OAS and the occurrence of stability hallucinations, indicating that this metric is a reliable indicator of alignment quality. By integrating OAS directly into the training process of models like CosyVoice2, the LLM is guided to learn more continuous and stable alignments between text and speech. This helps the model produce more accurate and stable speech outputs.
Attention-Guided Training for Enhanced Stability
Beyond the OAS, the paper introduces another powerful technique: attention-guided training. This method uses the attention values from a pre-trained, well-performing model (referred to as the “teacher” model) to guide the training of a new “student” model, also based on CosyVoice2. This is done using a “chain-of-thought” (CoT) approach, where the student model learns from the teacher’s established alignment patterns.
Instead of relying on difficult-to-obtain “forced alignment labels,” which explicitly map text to speech, this method uses the optimal alignment paths extracted from the teacher model’s attention as “pseudo forced alignment labels.” To make this guidance even more robust, two improvements were made. First, the predicted text tokens are not fed back into the next step, preventing error accumulation. Second, text tokens are used in a “sparse repetition” format, where each text token is marked only once in the target sequence, avoiding ambiguous boundary classifications. This provides more stable supervision for the student model.
Furthermore, to help the LLM track its progress through the text, especially with complex sentences, an additional prediction task for a “progress bar value” was introduced. This value provides absolute positional information, helping the model understand how much of the text has already been synthesized. Combining sparse text token supervision with progress bar values significantly reduces misalignment and further alleviates stability hallucinations.
Experimental Validation and Results
The proposed methods were rigorously tested on the CosyVoice2 model using extensive datasets, including WenetSpeech4TTS for training and Seed-TTS-Eval and CV3-Eval for evaluation. The experiments focused on both “hard text” scenarios (where hallucinations are common) and “common text” scenarios. The results were compelling: incorporating the OAS loss significantly reduced the Word Error Rate (WER) in synthesized speech, particularly for hard texts, without negatively impacting speech naturalness or expressiveness.
The attention-guided training, especially with sparse text token supervision and the progress bar value, further improved performance, leading to even lower WERs and more stable speech. Importantly, these improvements were achieved without requiring additional, often costly, forced alignment labels. The research paper can be found at this link.
Also Read:
- Z-Scores: A Diagnostic Tool for Improving Disfluency Removal in AI
- Understanding and Addressing Hallucinations in AI Agents
Conclusion
By deeply analyzing the attention mechanism in LLM-based TTS models and introducing the Optimal Alignment Score and attention-guided training, this research offers an effective solution to the persistent problem of stability hallucinations. The methods successfully reduce repetitive or omitted speech in challenging text scenarios, paving the way for more robust and natural speech synthesis systems.
