TLDR: ControlAudio is a new method for generating audio from text, offering precise control over timing and producing intelligible speech. It uses a progressive diffusion model, training in stages with specially constructed datasets and a “structured prompt” system that incorporates text, timing, and phoneme information. This allows it to create complex audio scenes with accurate event placement and clear spoken content, outperforming previous text-to-audio systems.
Researchers have introduced ControlAudio, a novel approach to text-to-audio (TTA) generation that offers unprecedented control over both the timing of sound events and the intelligibility of spoken content within generated audio. This new system addresses a significant challenge in the field: the difficulty of creating complex audio scenes with precise temporal accuracy and clear speech, especially due to the scarcity of suitable training data.
ControlAudio redefines controllable TTA generation as a multi-task learning problem, employing a progressive diffusion modeling strategy. This method allows the system to gradually learn and integrate fine-grained control signals, including text descriptions, specific timing information, and even phoneme features, through a step-by-step process.
Innovative Data Construction
One of the core innovations of ControlAudio lies in its approach to data. Recognizing the limitations of existing datasets, the team developed a dual strategy for data construction. They first processed existing annotated datasets, like AudioSet-SL, to extract clean speech tracks, segment them into individual events, and then transcribe these segments using advanced language models such as Gemini 2.5 Pro. This process transforms generic annotations into rich, content-specific events, for example, changing “man speaking” to “man speaking: ‘It’s been raining all day.'”
To further expand the training data, a large-scale simulated dataset was created. This involved analyzing real-world audio patterns to derive statistical priors on speech activity, then synthesizing monologues and dialogues by combining speech samples from LibriTTS-R with non-speech backgrounds from WavCaps and VGG-Sound. This simulation significantly boosts the scale and diversity of the training data, enabling the model to learn from a broader range of scenarios.
Unified Semantic Modeling with Structured Prompts
ControlAudio introduces a “Structured Prompt” (ys) to handle diverse control information—text, timing, and phonemes—using a single text encoder. This structured format uses special tokens to explicitly define audio events and their precise start and end times, overcoming the ambiguity often found in natural language descriptions. For instance, instead of a vague phrase like “an alarm sounds from low to high from 1 second to 9 seconds,” the structured prompt clearly delineates temporal boundaries.
For intelligible speech generation, the system leverages the explicit timing windows provided by the structured prompt. This allows the model to bypass the complex task of predicting phoneme or word durations, a common challenge in traditional text-to-speech (TTS) systems. By augmenting the encoder’s vocabulary with phoneme tokens, ControlAudio can render precise phonetic sequences within specified temporal boundaries, ensuring clear and accurate speech.
Progressive Model Training and Sampling
The training of ControlAudio unfolds in three progressive stages. Initially, a diffusion transformer (DiT) is pre-trained on vast text-audio datasets to establish a foundational understanding of text-guided audio generation. In the second stage, the model is fine-tuned with timing-annotated data, learning to interpret structured prompts for precise temporal control. The final stage involves joint training for both timing control and speech intelligibility, where the text encoder is unfrozen and optimized alongside the diffusion model using the comprehensive multi-source dataset.
During inference, ControlAudio employs a “Progressively Guided Sampling” strategy. This involves two phases: an initial phase that uses a simplified prompt and a low guidance scale to establish the overall temporal structure of audio events, followed by a second phase with a complete, phoneme-inclusive prompt and a higher guidance scale to ensure highly intelligible speech within the established structure. This coarse-to-fine approach significantly enhances both temporal accuracy and speech clarity.
Also Read:
- DiTSinger: Advancing Singing Voice Synthesis with Scalable Data and Implicit Alignment
- Unsupervised Speech Dereverberation: A Hybrid Model for Clearer Audio
State-of-the-Art Performance
Extensive experiments have shown that ControlAudio achieves state-of-the-art performance in controllable audio generation tasks. It significantly outperforms existing methods in objective and subjective evaluations for temporal accuracy and speech clarity. Crucially, it maintains high generative performance for general text-to-audio tasks, demonstrating that its fine-grained control capabilities do not compromise overall audio quality.
While ControlAudio marks a significant advancement, the researchers acknowledge limitations, including the current lack of control over stylistic speech attributes like emotion or speaker identity, and a potential trade-off between general audio quality and speech intelligibility in highly complex scenes. Future work aims to address these areas, further enhancing the model’s capabilities.
For more technical details, the full research paper can be accessed here: ControlAudio: Tackling Text-Guided, Timing-Indicated and Intelligible Audio Generation via Progressive Diffusion Modeling.
