TLDR: Ming-UniAudio is a new speech AI framework that unifies understanding, generation, and free-form editing of speech using a novel continuous tokenizer called MingTok-Audio. It addresses the challenge of conflicting representation needs for these tasks, achieving state-of-the-art performance in various benchmarks and enabling natural language-guided speech modifications without needing timestamps. The model and its components are open-sourced to encourage further research.
In a significant advancement for artificial intelligence, researchers from Inclusion AI and Ant Group have introduced Ming-UniAudio, a groundbreaking speech large language model (LLM) designed to unify speech understanding, generation, and editing. This innovative framework addresses a long-standing challenge in speech AI: the conflicting demands of token representations for understanding and generation tasks, which previously hindered instruction-based free-form speech editing.
At the heart of Ming-UniAudio is MingTok-Audio, a novel unified continuous speech tokenizer. This is the first continuous tokenizer that effectively integrates both semantic (meaning-related) and acoustic (sound-related) features, making it equally suitable for tasks that require comprehending spoken language and those that involve creating it. Traditional speech models often had to compromise, either using separate representations for understanding and generation, which made editing difficult, or relying on discrete tokens that lost fine-grained speech details.
The Ming-UniAudio model, built upon this unified tokenizer, strikes a crucial balance between generation and understanding capabilities. It has already set new state-of-the-art (SOTA) records on 8 out of 12 metrics on the ContextASR benchmark, demonstrating its superior ability to understand speech in context. For speech generation, it achieves a highly competitive Seed-TTS-WER of 0.95 for Chinese voice cloning, indicating excellent speech intelligibility.
One of the most exciting aspects of this research is the development of Ming-UniAudio-Edit, a dedicated speech editing model. This is the first speech LLM that enables universal, free-form speech editing guided solely by natural language instructions. This means users can simply tell the model what changes they want to make, whether it’s modifying semantic content (like inserting, deleting, or substituting words) or adjusting acoustic attributes (such as denoising, changing speed, pitch, or emotion), without needing to specify exact timestamps. This capability opens up new possibilities for intuitive and flexible audio manipulation.
To rigorously evaluate these new editing capabilities and provide a foundation for future research, the team also introduced Ming-Freeform-Audio-Edit. This is the first comprehensive benchmark specifically designed for instruction-based free-form speech editing, covering diverse scenarios and evaluating semantic correctness, acoustic quality, and how well the model follows instructions.
The development of Ming-UniAudio involved a sophisticated three-stage training process for its tokenizer, focusing on acoustic reconstruction, semantic feature distillation, and unified tokenizer training with an LLM. This meticulous approach ensures that the model’s unified representation is rich in both semantic and acoustic information, crucial for its versatile performance.
The researchers have open-sourced the continuous audio tokenizer, the unified foundational model, and the free-form instruction-based editing model. This move aims to foster further development in unified audio understanding, generation, and manipulation within the broader AI community. You can find more details about this work in the full research paper: Ming-UniAudio: Speech LLM for Joint Understanding, Generation and Editing with Unified Representation.
Also Read:
- Protecting Voices from AI Cloning: E2E-VGuard’s Dual Defense Against Advanced Speech Synthesis
- MERaLiON-SER: Advancing Speech Emotion Recognition for Global Languages
This work represents a significant step towards creating more intelligent and intuitive human-machine interactions, where speech can be understood, generated, and edited with unprecedented flexibility and quality.
