TLDR: NExT-OMNI is an open-source omnimodal foundation model that unifies any-to-any cross-modal understanding, generation, and retrieval using discrete flow matching. It overcomes limitations of autoregressive models by integrating understanding and generation capabilities more efficiently, leading to superior performance in multi-turn interactions and cross-modal retrieval, while also offering faster response times.
Artificial intelligence is rapidly evolving, and the dream of creating AI systems that can understand and generate information across all modalities – text, images, video, and audio – is closer than ever. This is the vision behind NExT-OMNI, a groundbreaking open-source omnimodal foundation model that aims to revolutionize how AI interacts with the world.
Authored by Run Luo, Xiaobo Xia, Lu Wang, Longze Chen, Renke Shan, Jing Luo, Min Yang, and Tat-Seng Chua, NExT-OMNI tackles a critical challenge in AI: the limitations of current multimodal models. Many existing models struggle with a balanced integration of understanding and generation capabilities, often relying on complex, separate designs for different tasks. This can lead to inefficiencies and restrict their use in broader scenarios like cross-modal retrieval.
A Unified Approach with Discrete Flow Matching
NExT-OMNI introduces a novel approach by leveraging “discrete flow matching” paradigms. Imagine a smooth, continuous path that transforms noisy, corrupted data into clear, meaningful information. Discrete flow matching provides a way for the model to learn these paths, enabling it to process and generate data across various modalities in a highly efficient and unified manner. This means NExT-OMNI can natively support any-to-any understanding and generation, from turning text into speech to generating video from an image, and vice-versa.
Unlike traditional autoregressive models that process information sequentially, NExT-OMNI starts with corrupted data and iteratively refines it in parallel. This “denoising” process allows for richer, bidirectional information integration, significantly enhancing performance. The model achieves this through concise, unified representations of different data types, rather than relying on separate, task-specific designs.
Enhanced Capabilities and Efficiency
Trained on vast amounts of interleaved text, image, video, and audio data, NExT-OMNI demonstrates impressive capabilities. It delivers competitive performance on standard multimodal generation and understanding benchmarks. More importantly, it significantly outperforms prior unified models in complex multi-turn multimodal interactions and cross-modal retrieval tasks. This highlights a key architectural advantage: its ability to deeply fuse multimodal features and process them with enhanced response efficiency.
The model’s design also incorporates clever strategies for improved performance and speed. A “dynamic length generation” strategy helps it adapt to varying response lengths in understanding tasks, while an “adaptive caching” mechanism accelerates inference by reusing stable features. These innovations contribute to NExT-OMNI’s superior performance and faster response times compared to many existing AI architectures.
Also Read:
- RAG-Anything: A Unified Approach to Multimodal Information Retrieval
- OmniVideoBench: A New Benchmark for Advanced Audio-Visual AI Understanding
Open-Source for Future Innovation
A significant aspect of NExT-OMNI is its commitment to open-source development. The researchers are releasing training details, data protocols, code, and model checkpoints. This move is crucial for fostering further research and accelerating the development of next-generation multimodal foundation models. The paper suggests that unified omnimodal models like NExT-OMNI could serve as a “world brain,” continuously evolving through interactions with real-world multimodal data, ultimately contributing to the realization of Artificial General Intelligence (AGI).
For more in-depth technical details, you can refer to the full research paper available at arXiv:2510.13721.
