TLDR: DurFormer is a novel adaptive duration model for text-to-speech (TTS) alignment that addresses common issues like word skipping or repetition in synthesized speech. It improves alignment accuracy and robustness by incorporating subjective factors (speaking speed), objective factors (speaking scenarios), and semantic meaning from the text. Crucially, it predicts phoneme durations as a probability distribution (mean and variance) rather than a fixed value, leading to more natural and stable speech synthesis, outperforming previous baseline models.
Neural text-to-speech (TTS) models are designed to convert written text into natural-sounding speech. While these models have made significant strides, especially in generating speech for familiar types of text, they often face challenges when dealing with longer sentences or text that is outside their usual training data. This can lead to issues like words being skipped or repeated in the synthesized speech.
A crucial part of any TTS system is the alignment mechanism, which links the text (linguistic information) to the sound (acoustic representation). Traditional autoregressive TTS models, which generate speech sequentially, often use an attention mechanism to learn these alignments on the fly. However, these alignments can be unstable and struggle to adapt to new situations.
On the other hand, non-autoregressive TTS models, which can generate speech in parallel, often need explicit information about how long each sound (phoneme) should last. This ‘duration’ information is usually extracted from external sources, requiring additional models. Accurate phoneme-level duration modeling is vital for creating speech with natural rhythm and prosody, directly impacting how good the synthesized speech sounds.
To address these limitations, researchers have introduced a new framework called DurFormer, an adaptive duration model for text-speech alignment. This innovative model aims to provide more precise and adaptable phoneme-level duration predictions, making TTS models, especially non-autoregressive ones, more robust and natural-sounding. You can read the full research paper here: ADAPTIVE DURATION MODEL FOR TEXT SPEECH ALIGNMENT.
How DurFormer Works
DurFormer approaches duration prediction as a conditional regression problem, meaning it predicts durations based on various influencing factors. The model is designed with several key components:
- Attribute Encoder: This component recognizes that phoneme durations are influenced by both subjective and objective factors. Subjective factors relate to the speaker’s habits, such as their speaking speed. DurFormer categorizes speaking speed into five levels: very slow, slow, moderate, fast, and very fast. Objective factors refer to external conditions, like the speaking scenario (e.g., a formal presentation versus a casual conversation). The Attribute Encoder quantifies and integrates these factors into the prediction process.
- Semantic Fusion Module: The meaning and emotion conveyed by the text also affect speech rhythm and phoneme durations. This module takes the text script and processes it through a pre-trained language model to extract its underlying semantic meaning. This semantic information is then fused into the duration prediction, ensuring the synthesized speech reflects the linguistic nuances of the input text.
- Probability Module: Unlike previous models that predict a single, fixed duration value, DurFormer’s Probability Module treats duration as a flexible, diverse element. It predicts not just the average duration (mean) but also the variability (variance) of the duration sequence. This probabilistic approach allows the model to generate more robust and expressive alignments, leading to more natural rhythm and timing in the synthesized speech.
Experimental Results and Impact
Experiments conducted using a subset of the WenetSpeech4TTS dataset showed that DurFormer significantly outperforms baseline models like FastSpeech and FastSpeech2 in alignment accuracy. Numerically, it demonstrated roughly an 11.3 percent improvement in alignment accuracy, indicating more precise predictions. Furthermore, DurFormer proved to be more efficient than other popular methods, such as diffusion-based flow-matching models, achieving superior performance with a much smaller model size.
The impact of DurFormer on the overall quality of synthesized speech was also evaluated. When integrated into existing TTS systems, DurFormer provided more stable duration predictions, which in turn led to more consistent word error rates in the synthesized speech. This stability is particularly beneficial when the reference audio prompt for a TTS model differs significantly from the input text’s context.
Also Read:
- Enhancing 3D Facial Animation with Context-Aware Speech Modeling
- New Benchmarks in ASR for Impaired Speech: Insights from the Interspeech 2025 Challenge
Conclusion
DurFormer represents a significant advancement in text-to-speech alignment. By adaptively considering various influencing factors and predicting duration as a distribution rather than a fixed value, it offers more precise, versatile, and robust phoneme-level duration information. This innovation is expected to greatly enhance the performance and naturalness of neural text-to-speech systems, especially those that are non-autoregressive.
