TLDR: A new method called TKTO (Token-level Kahneman-Tversky Optimization) significantly improves Text-to-Speech (TTS) accuracy, especially for ambiguous languages like Japanese. It achieves this by optimizing at the individual token level and eliminating the need for scarce paired desirable/undesirable data, making training more data-efficient. TKTO boosts Japanese TTS accuracy by 39% and reduces Character Error Rate by 54%, outperforming existing models.
Text-to-Speech (TTS) technology has seen remarkable advancements, with models now capable of generating highly natural and robust speech. However, a significant challenge remains in fine-tuning these systems to perfectly align with human preferences, especially for complex languages like Japanese.
Traditional methods for improving TTS quality often rely on “preference optimization,” where models learn from human feedback. A common technique, Direct Preference Optimization (DPO), requires pairs of desirable and undesirable speech samples for the same sentence. This approach faces two major hurdles: first, such paired data is scarce and expensive to collect, leading to inefficient training. Second, these methods typically optimize at the “utterance level,” meaning they look at the entire sentence. However, pronunciation errors often occur at a much finer “token level” – individual characters or words – and utterance-level optimization can miss these subtle but crucial details.
Consider Japanese, for instance. A single word can have different pronunciations and meanings depending on the context. The paper highlights the word “辛い” which can mean “spicy” (karai) or “hard” (tsurai). Current systems, even those using advanced Grapheme-to-Phoneme (G2P) converters or Large Language Model (LLM)-based TTS, can struggle with such ambiguities, leading to incorrect pronunciations.
Introducing TKTO: A Novel Approach
To tackle these challenges, researchers Rikuto Kotoge and Yuichi Sasaki propose a new framework called Token-level Kahneman-Tversky Optimization (TKTO). This innovative method offers two key advantages:
- It eliminates the need for paired desirable and undesirable data, making the training process far more data-efficient. Instead, it can learn from unpaired samples, which are much easier to obtain.
- It directly targets token-level units, automatically providing precise alignment signals without requiring manual token-level annotations. This allows for fine-grained optimization exactly where it’s needed for accurate pronunciation.
TKTO is inspired by Kahneman-Tversky’s prospect theory and operates in two main steps. First, it estimates the importance of each token. This is done by constructing two contrasting LLMs: one that favors high-reward tokens and another that favors low-reward tokens. By comparing these, TKTO assigns a “weight” to each token, indicating its significance for preference learning. Second, it uses these token-level importance weights to guide a refined optimization process, extending the concept of KTO (Kahneman-Tversky Optimization) to the token level.
Also Read:
- TokenChain: Unlocking Efficient Speech AI with Discrete Semantic Tokens
- Bridging the Gap Between Speech and Text Models with Latent Speech Patches
Impressive Results and Future Implications
The effectiveness of TKTO was demonstrated through extensive evaluations, particularly on the challenging task of ambiguous Japanese pronunciation. The results are compelling:
- TKTO improved Japanese TTS accuracy by a remarkable 39%.
- It reduced the Character Error Rate (CER) by 54%.
- The system automatically assigned 12.8 times stronger rewards to the targeted tokens, confirming its ability to focus on critical pronunciation units.
When compared against various baselines, including industry-leading models like gpt-4o-mini-tts and gemini-2.5-tts-preview, as well as other preference optimization methods, TKTO consistently achieved superior performance in terms of accuracy, CER, and the ratio of “bad cases” (samples with high error rates). Notably, its ability to leverage unpaired data was a significant factor in surpassing these strong benchmarks.
Subjective evaluations, including naturalness mean opinion scores (NMOS) and ABX preference tests, further confirmed that speech generated by TKTO was perceived as more natural and preferred over outputs from base models and standard KTO.
While the current evaluations focused on Japanese, the underlying TKTO framework holds promise for application in other text generation tasks where specific tokens play a crucial role. This research marks a significant step towards more robust, natural, and data-efficient LLM-based TTS systems. You can read the full research paper for more details here.
