TLDR: HiStyle is a new method for controllable speech synthesis that uses a two-stage predictor to generate expressive speech from text prompts. It’s based on the discovery that speech style embeddings have a hierarchical structure (timbre first, then specific attributes). The system also uses contrastive learning and an improved data annotation strategy, leading to better control over speech style and high-quality, natural-sounding speech.
Controllable speech synthesis, a key technology in digital content creation and human-computer interaction, has seen significant advancements with the integration of large language models (LLMs) and diffusion models. These innovations have shifted text-to-speech (TTS) systems from relying on predefined labels to using natural language descriptions for controlling speaking style. However, a common challenge in these systems is that they often overlook the intricate distribution of style embeddings, which are crucial for precisely manipulating attributes like gender, volume, speech rate, pitch, and pitch fluctuation.
Researchers have now introduced HiStyle, a novel approach that addresses this limitation by leveraging a newly discovered hierarchical clustering pattern within global style embeddings. Through detailed analysis, it was observed that these embeddings first group by timbre (the unique quality of a voice) and then further subdivide into finer clusters based on specific style attributes.
Inspired by this insight, HiStyle proposes a two-stage style embedding predictor. This system first predicts coarse-grained, global speaker-related embeddings and then refines these into fine-grained style embeddings, all conditioned on textual prompts. To further enhance the alignment between text descriptions and the resulting audio, HiStyle incorporates contrastive learning during its training process. This helps the system better understand and connect the textual and acoustic representations of style.
Beyond the prediction model, HiStyle also introduces an innovative style annotation strategy. This strategy combines objective statistical methodologies with subjective human auditory preferences to create more accurate and perceptually consistent textual prompts for style control. This dual approach ensures that the labels used for training are both technically sound and align with how humans perceive speech styles.
The HiStyle system works by taking a text description and processing it through a text prompt encoder. This is followed by a speaker embedding predictor, which captures global speaker information. Finally, a style embedding predictor refines this into the ultimate style embedding. Both predictors are built using conditional diffusion models based on transformer blocks, which are adept at handling complex data distributions.
Extensive experiments have demonstrated that HiStyle significantly improves style controllability when applied to a base TTS model. It outperforms alternative style embedding prediction methods across various objective metrics, achieving higher accuracy in controlling attributes like volume, pitch, and fluctuation. Crucially, HiStyle maintains high speech quality in terms of naturalness and intelligibility, ensuring that the enhanced control does not compromise the overall listening experience. Subjective evaluations also confirm HiStyle’s superior style consistency and competitive naturalness compared to other systems.
An ablation study further highlighted the importance of both contrastive learning and the advanced style annotation strategy. Removing either component led to a noticeable decline in performance, underscoring their complementary roles in HiStyle’s success.
Also Read:
- Unlocking Personalized AI: Learning Directly from User Conversations
- Enhancing In-Game Voice Communication Accuracy with a New AI Framework
In conclusion, HiStyle represents a significant step forward in text-prompt-guided controllable speech synthesis. By recognizing and utilizing the hierarchical structure of style embeddings, integrating contrastive learning, and employing a sophisticated data annotation pipeline, HiStyle paves the way for more versatile and user-friendly speech synthesis systems that offer precise control over expressive speech. You can read the full research paper for more details. Read the full research paper here.
