TLDR: Emotion-Aware Stepwise Preference Optimization (EASPO) is a new post-training framework for diffusion text-to-speech models that significantly enhances emotional expressiveness and naturalness. It introduces an Emotion-aware Stepwise Preference Model (EASPM) to provide fine-grained, time-conditioned feedback at intermediate denoising steps. By optimizing generation based on ‘win-lose’ preference pairs at each step, EASPO overcomes the limitations of sparse, utterance-level feedback, leading to superior performance in generating emotionally consistent and high-quality synthetic speech.
The field of artificial intelligence continues to advance at a rapid pace, bringing us closer to more natural and human-like interactions with machines. One significant area of development is Emotional Text-to-Speech (Emo-TTS), which aims to generate speech that not only sounds intelligible but also conveys a wide range of human emotions and prosodic nuances. This capability is crucial for creating more engaging conversational agents, improving accessibility, and enhancing content creation.
However, achieving fine-grained emotional control in synthesized speech without sacrificing naturalness has been a persistent challenge. Existing methods often rely on broad, coarse labels for emotions or use proxy classifiers, and the feedback they receive is typically limited to the entire utterance. This sparse feedback makes it difficult to guide the model in capturing the subtle, time-evolving emotional cues that are essential for truly expressive speech.
Introducing Emotion-Aware Stepwise Preference Optimization (EASPO)
A new post-training framework, Emotion-Aware Stepwise Preference Optimization (EASPO), has been introduced to address these limitations. Developed by Jiacheng Shi, Hongfei Du, Yangfan He, Y. Alicia Hong, and Ye Gao, EASPO is designed to align diffusion Text-to-Speech (TTS) models with detailed emotional preferences at the intermediate stages of the denoising process. This innovative approach moves beyond endpoint preferences and categorical labels, offering a more granular level of supervision.
Central to EASPO is the Emotion-aware Stepwise Preference Model (EASPM). This time-conditioned model plays a crucial role by scoring noisy intermediate speech states. Imagine the process of generating speech as a series of steps, where noise is gradually removed to reveal the final audio. At each of these steps, EASPM evaluates multiple candidate speech segments, assessing their emotional expressiveness. It then identifies a ‘win-lose’ pair – one candidate that is emotionally preferred and another that is less so, yet both maintain similar linguistic content and subtle prosodic differences.
EASPO leverages these stepwise preferences to optimize the speech generation. Instead of propagating a single preference across all diffusion steps, EASPO focuses on local, time-conditioned tasks. This means that at each denoising step, the model learns to steer the generation towards more emotionally consistent speech by minimizing the difference between the preferred and dis-preferred samples. After selecting a win-lose pair for optimization, the system randomly chooses one candidate to continue the generation process, preventing bias and encouraging diverse speech trajectories.
How EASPO Works
The framework builds upon a pre-trained Grad-TTS model and introduces dense, emotion-aligned rewards at each denoising step. EASPM, adapted from CLEP (a contrastive audio-language encoder), is fine-tuned on large-scale emotional speech data to accurately score the emotional expressiveness and prosody of noisy intermediate states. This allows for dense, temporally grounded preference learning and on-the-fly scoring.
The objective function of EASPO extends Direct Preference Optimization (DPO) by integrating these stepwise preference signals. It updates the diffusion policy so that the log-likelihood ratio difference between the ‘win’ and ‘lose’ transitions matches the dense emotional reward difference provided by EASPM. This method has been shown to yield more stable reward optimization compared to standard policy gradient methods.
Experimental Validation and Superior Performance
The researchers conducted extensive experiments, fine-tuning EASPM on the English MSP-Podcast corpus and evaluating the full EASPO framework on the English split of the ESD dataset. They compared EASPO against seven other emotion-controllable TTS baselines, including FG-TTS, PromptTTS, Emospeech, EmoDiff, CosyVoice, CosyVoice2, and EmoVoice.
The results were compelling. EASPO demonstrated superior performance across various objective metrics, including emotion similarity, prosody similarity, intelligibility (measured by Word Error Rate or WER), and perceptual quality (UTMOS). Notably, EASPO achieved a 2.07% gain in Emotion Similarity over CosyVoice. Subjective evaluations by human raters also confirmed EASPO’s advantage in naturalness, emotional expressiveness, emotion consistency, and emotion recall.
Ablation studies further validated the importance of EASPO’s key components, such as timestep conditioning in EASPM and the random selection strategy for initializing the next iteration. These studies confirmed that each element contributes significantly to the framework’s overall effectiveness in generating emotionally aligned speech with enhanced coherence, while preserving natural prosody and speech quality.
Also Read:
- HiStyle: Enhancing Speech Synthesis with Hierarchical Style Prediction
- Emo-TTA: Enhancing Speech Emotion Recognition in Dynamic Environments
Conclusion
EASPO represents a significant step forward in emotional text-to-speech synthesis. By introducing stepwise preference optimization and an emotion-aware scoring model, it enables fine-grained emotional alignment throughout the speech generation process. This allows diffusion models to capture subtle affective cues more effectively, leading to the creation of highly expressive and natural-sounding artificial voices. For more details, you can read the full research paper here.
