TLDR: LM-based Text-to-Speech (TTS) models often generate ‘hallucinated’ speech that deviates from input text. This research introduces GOAT (GFlOwNet-guided distribution AlignmenT), a novel post-training framework that mitigates these hallucinations. GOAT reformulates TTS generation as a trajectory flow optimization problem using GFlowNets, aligning the model’s output distribution towards high-confidence sequences. It significantly reduces character error rates (over 50%) and model uncertainty (up to 58%) without requiring massive training resources or introducing significant inference latency, demonstrating strong generalization across languages.
Text-to-Speech (TTS) technology has made incredible strides, allowing computers to convert written text into natural-sounding speech. These systems are crucial for many applications, from virtual assistants to accessibility tools. However, a persistent challenge in advanced Language Model (LM)-based TTS systems is the phenomenon of “hallucinations.”
Understanding Hallucinations in AI Speech
In the context of AI, hallucinations refer to instances where a model generates content that deviates from the intended input or factual information. For LM-based TTS models, this means the generated speech might not accurately reflect the input text. This can manifest as mispronunciations, missing words, or even semantic inconsistencies, especially when dealing with longer or more complex sentences. Imagine asking a system to read a sentence, and it skips a word or pronounces it incorrectly – that’s a hallucination.
Current methods to combat these hallucinations often come with significant drawbacks. Some require vast amounts of training data and computational power, making them expensive and resource-intensive. Others introduce delays during the speech generation process, which is problematic for real-time applications.
Introducing GOAT: A New Approach to Mitigate Hallucinations
A recent research paper, titled “Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets,” introduces a novel framework called GFlOwNet-guided distribution AlignmenT (GOAT). This framework offers a promising solution to reduce hallucinations without demanding excessive training resources or introducing significant delays in speech generation. You can read the full paper here: Mitigating Hallucinations in LM-Based TTS Models via Distribution Alignment Using GFlowNets.
The authors, Chenlin Liu, Minghui Fang, Patrick, Wei Zhou, Jie Gao, and Jiqing Han, first conducted an in-depth analysis of LM-based TTS models. They discovered a strong connection between hallucinations and the model’s uncertainty during the speech generation process. Essentially, when the model is less certain about the next speech token to generate, it’s more likely to make an error.
How GOAT Works
Based on this insight, GOAT redefines the TTS generation process as an optimization problem, guiding the model to find more reliable and optimal paths for generating speech. It leverages a concept called GFlowNets, which are a type of generative model designed to learn a distribution over complex objects, in this case, speech token sequences.
Key components of GOAT include:
- An enhanced Sub-trajectory Balance objective: This helps the model learn from various lengths of speech segments, preventing fragmented errors.
- A sharpened internal reward: GOAT uses the model’s own token sampling probabilities as a reward signal, encouraging it to favor high-quality speech sequences. A special “reward temperature decay” strategy is used to balance performance and training stability.
- Learning rate optimization: This ensures a stable training process, preventing the model from learning undesirable shortcuts that could lead to more hallucinations.
Also Read:
- FINCH-ZK: Enhancing Factual Reliability in Large Language Models
- Advancing Speaker Tracking for Real-Time Applications with Short-Context Embeddings
Impressive Results and Broad Applicability
Extensive experiments using CosyVoice2, a standard LM-based TTS architecture, demonstrated GOAT’s effectiveness. The framework reduced character error rates by over 50% on challenging test cases, significantly improving the accuracy of generated speech. It also lowered model uncertainty by up to 58%, confirming the link between uncertainty and hallucinations.
One of GOAT’s most compelling features is its strong generalization ability. It showed consistent improvements even when trained and evaluated on different languages (Chinese and English) or mixed-language datasets. Crucially, GOAT achieves these improvements without adding significant inference latency, meaning it doesn’t slow down the speech generation process.
In conclusion, GOAT offers an inspiring solution for making LM-based TTS models more reliable and less prone to generating incorrect speech. By focusing on distribution alignment and reducing model uncertainty, this framework paves the way for higher-quality, more consistent AI-generated voices across various applications.
