TLDR: DiTSinger is a new Singing Voice Synthesis (SVS) framework that addresses data scarcity and model scalability challenges. It uses a two-stage pipeline to generate over 500 hours of high-quality singing data by combining LLM-generated lyrics with melody-specific models. The core of DiTSinger is a Diffusion Transformer with RoPE and qk-norm, scaled for enhanced fidelity. A key innovation is its implicit alignment mechanism, which removes the need for phoneme-level duration labels by constraining phoneme-to-acoustic attention within character-level spans, improving robustness. Experiments show DiTSinger’s strong scalability and superior performance compared to state-of-the-art SVS methods.
Singing Voice Synthesis (SVS), the technology that generates singing from lyrics and musical scores, has seen significant advancements. However, it still faces considerable hurdles, primarily related to the scarcity of high-quality training data and the challenge of scaling models effectively. Traditional SVS methods often struggle with precise phoneme-pitch alignment and require extensive, manually annotated datasets, which are costly and time-consuming to produce.
A new research paper introduces DiTSinger, a novel framework designed to overcome these limitations. DiTSinger leverages a Diffusion Transformer architecture and an innovative implicit alignment mechanism to achieve scalable, alignment-free, and high-fidelity singing voice synthesis. The core idea is to make SVS models more robust and capable of generating diverse and natural-sounding singing voices without relying on detailed phoneme-level duration labels.
A Two-Stage Data Generation Pipeline
One of DiTSinger’s key innovations is its two-stage data construction pipeline, which addresses the data scarcity problem. This pipeline is illustrated in Figure 1 of the research paper. The first stage, called the Recording-fitting Phase, involves creating a compact dataset. Here, a small set of fixed melodies is chosen, and a large language model (LLM) generates diverse lyrical variations for these melodies. Professional singers then record these lyrics, resulting in high-quality vocal recordings. These recordings are used to train melody-specific SVS models, referred to as PseudoSinger.
In the second stage, the Data Expansion Phase, the trained PseudoSinger models are put to work. They synthesize large-scale singing data by continuously generating new lyrics with the LLM and rendering them into singing voices. This approach allows for the creation of a vast dataset with consistent melodies but diverse lyrical content, significantly enhancing phonetic coverage and enabling controllable data augmentation. This method has successfully generated over 500 hours of high-quality Chinese singing data, forming one of the largest publicly reported SVS datasets.
DiTSinger Architecture: Diffusion Transformer with Enhancements
DiTSinger’s architecture is built upon a Diffusion Transformer (DiT), a powerful model known for its scalability and performance in generative tasks. The model predicts noise in the mel-spectrogram domain at each denoising step, conditioned on both fine-grained and coarse-grained inputs. Fine-grained conditions include pitch, phonemes, word durations, and slur indicators, while coarse-grained inputs cover speaker identity and diffusion timestep.
The DiTSinger model incorporates several advanced components to boost its performance. It uses Rotary Positional Encoding (RoPE) and QK-Norm within its Multi-Head Self-Attention layers, which are crucial for handling sequential data like audio. Adaptive Layer Normalization (AdaLN) is also employed, conditioned on speaker embeddings, to further refine the synthesis process.
Implicit Alignment Mechanism: A Game Changer
Perhaps the most significant architectural contribution is the Implicit Alignment Mechanism. Traditional SVS models often require explicit phoneme-level duration labels, which are difficult and expensive to obtain. DiTSinger bypasses this need by constraining each phoneme’s attention to its character-level temporal span. This means that instead of precise phoneme timings, the model uses the known start time and duration of each character, with a small tunable offset.
This mechanism creates a soft supervision, guiding the model to learn localized alignments under coarse timing constraints. During training, it relies solely on the diffusion reconstruction loss. At inference time, it enforces these temporal constraints, ensuring stable and consistent attention patterns. This approach significantly improves robustness, especially when dealing with noisy or uncertain alignments, and removes a major bottleneck in SVS data preparation.
Experimental Validation and Scalability
Extensive experiments validate DiTSinger’s effectiveness. The model demonstrates strong scalability across both model size and dataset scale. Larger models and more training data consistently lead to improved synthesis quality, as measured by objective metrics like MCD, FFE, and F0RMSE, and subjective Mean Opinion Score (MOS) tests.
The PseudoSinger data generation pipeline also proved highly effective. Experiments showed that using a moderate number of PseudoSinger groups (around 20) balanced specialization and generalization, leading to optimal performance. When compared against state-of-the-art SVS models like DiffSinger, StyleSinger, and TCSinger, DiTSinger L2 achieved superior overall performance, yielding the highest MOS and significantly lower error rates. This highlights the power of its implicit alignment framework and scalable architecture.
Also Read:
- Unsupervised Speech Dereverberation: A Hybrid Model for Clearer Audio
- Robust Emotion Recognition in Speech: Disentangling Features and Aligning Embeddings
Future Directions
While DiTSinger represents a significant leap forward, the current research acknowledges limitations. Experiments were primarily conducted on Chinese datasets, and the model does not yet explicitly account for complex singing techniques. Future work aims to expand datasets to support multilingual and multi-scenario adaptability and incorporate conditions such as reference timbre and singing style for even greater control and realism.
