TLDR: MeanAudio is a new text-to-audio generation model that uses Mean Flows and a unique training curriculum to achieve state-of-the-art audio quality with up to 100x faster inference speed compared to existing diffusion-based systems, while also being more parameter-efficient.
The paper introduces MeanAudio, a new model designed for generating audio from text prompts quickly and accurately. Text-to-Audio (TTA) generation is a rapidly advancing field with many applications, including virtual reality, gaming, film production, and human-computer interaction. While existing TTA systems offer high-quality audio and good control, their slow inference speed has been a major limitation for practical use.
MeanAudio addresses this challenge by utilizing a novel approach based on “Mean Flows.” Unlike traditional diffusion or flow-based models that require many iterative steps, MeanAudio learns to directly map from the starting point to the end point of a flow trajectory. This allows for very fast audio generation, often in a single step. The model is built on a “Flux-style latent transformer,” which is a type of neural network architecture.
A key innovation in MeanAudio is its ability to incorporate “classifier-free guidance” (CFG) directly into its training process. This means that the model learns to generate audio that aligns well with the text prompt without needing to perform extra calculations during the generation phase, which typically doubles the inference cost in other systems. This integration helps maintain strong prompt alignment without sacrificing speed.
To ensure stable and efficient training, the researchers developed a unique “instantaneous-to-mean curriculum” combined with “flow field mix-up.” This training strategy first teaches the model the basic, instantaneous dynamics of audio generation, and then gradually adapts it to learn the more complex “mean flows.” This two-stage approach is crucial for improving both the training efficiency and the quality of the generated audio.
Experimental results highlight MeanAudio’s impressive performance. In single-step audio generation, it achieves state-of-the-art results across various metrics. Notably, it boasts a real-time factor (RTF) of 0.013 on an NVIDIA RTX 3090 GPU, which translates to a remarkable 100 times speedup compared to leading diffusion-based TTA systems. Even with multi-step generation, MeanAudio maintains strong performance, producing smooth and coherent audio transitions.
The model is also quite efficient in terms of size, with only 120 million parameters, which is significantly smaller than many other TTA systems that often exceed 500 million parameters. This makes MeanAudio more accessible for training on consumer-grade GPUs. The researchers have committed to open-sourcing all training codes and model weights to encourage further research in this area.
Also Read:
- Advancing Speech Diarization and Separation Without Prior Enrollments
- SpeakerLM: An End-to-End AI Solution for Speaker Diarization and Recognition
This breakthrough in text-to-audio generation promises to unlock new possibilities for real-time applications and creative workflows, making high-quality audio synthesis more practical and accessible. For more technical details, you can refer to the full research paper available at MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows.
