TLDR: LongCat-Flash-Omni is a 560-billion-parameter open-source omni-modal AI model developed by the Meituan LongCat Team. It excels in real-time audio-visual interaction by employing a curriculum-inspired progressive training strategy and an efficient Shortcut-connected Mixture-of-Experts (MoE) architecture. The model achieves state-of-the-art performance across text, image, video, and audio understanding and generation, while maintaining low-latency responses through a modality-decoupled parallelism training infrastructure and an asynchronous streaming inference pipeline. It aims to unify robust offline multimodal understanding with real-time human-AI interaction.
The Meituan LongCat Team has unveiled LongCat-Flash-Omni, a groundbreaking open-source omni-modal model designed to excel in real-time audio-visual interaction. This advanced model, boasting 560 billion parameters, represents a significant step towards Artificial General Intelligence (AGI) by seamlessly integrating diverse forms of information, much like humans do.
Addressing Key Challenges in Multimodal AI
Developing an AI that can understand and interact across multiple senses in real-time presents several complex challenges. The LongCat-Flash-Omni project specifically tackles four major hurdles:
- Cross-modal heterogeneity: Different types of data (text, audio, visual) have vastly different structures. The model needs to find unified ways to represent and combine these, ensuring that performance in one area doesn’t suffer when combined with others.
- Unified offline and streaming capabilities: The model must handle both traditional, pre-processed data and live, continuous streams of information, which requires distinct processing abilities like understanding relative time and precise synchronization.
- Real-time interaction: Achieving low-latency responses for streaming audio and video input, as well as generating speech output, demands highly efficient model architecture and deployment infrastructure.
- Training Efficiency: The sheer size and diverse nature of multimodal data make distributed training incredibly complex and resource-intensive.
Innovative Solutions and Architecture
LongCat-Flash-Omni addresses these challenges through a meticulously designed multi-stage training pipeline and an efficient architecture. It builds upon LongCat-Flash, which uses a high-performance Shortcut-connected Mixture-of-Experts (MoE) architecture, allowing it to activate only a portion of its parameters (around 27 billion out of 560 billion) for efficiency.
The model integrates efficient multimodal perception and speech reconstruction modules. A vision encoder, called LongCat-ViT, processes images and videos at their native resolutions, avoiding information loss. An audio tokenizer, encoder, and decoder handle speech, converting raw audio into discrete tokens for the model and reconstructing waveforms for speech output. The core of the system is a large language model (LLM) backbone that processes these multimodal inputs and generates responses.
For real-time interaction, the model employs a chunk-wise interleaving mechanism, where audio and visual features are synchronized and fed into the LLM as quickly as possible to minimize response latency. A sparse-dense sampling strategy helps balance computational cost and information retention during user interactions.
Progressive Training and Data Strategies
The training of LongCat-Flash-Omni follows a curriculum-inspired, progressive strategy, moving from simpler to more complex tasks. It starts with extensive text pre-training, then gradually incorporates speech, image, and video data. This ensures a strong foundation in each modality while fostering deep integration across them. The model’s context window is also extended to an impressive 128K tokens, enabling advanced long-term memory and multi-turn dialogue capabilities.
A significant effort was put into data curation, collecting a diverse corpus of over 2.5 trillion tokens, including speech-text interleaved data, generic image-text data, OCR, grounding, GUI data, STEM data, multi-image data, video data, and specialized long-context multimodal data. Post-training involves Supervised Fine-Tuning (SFT) with high-quality instruction data and Reinforcement Learning (RL) using Direct Preference Optimization (DPO) to align the model’s behavior with human preferences and ensure coherent text and speech outputs.
Efficient Infrastructure and Deployment
To manage the immense scale and heterogeneity, the team developed a modality-decoupled parallelism (MDP) training scheme. This innovative approach allows independent optimization of the LLM, vision encoder, and audio encoder, sustaining over 90% of the throughput achieved by text-only training. For deployment, a decoupled inference framework and an asynchronous streaming pipeline ensure low-latency, real-time audio-visual interaction, allowing users to receive responses within milliseconds after their input.
Also Read:
- Unifying Human Motion, Vision, and Language Understanding with a New AI Framework
- Distributing Intelligence: How Networked AI Experts Power Mobile Devices
Leading Performance and Open-Source Contribution
Extensive evaluations demonstrate that LongCat-Flash-Omni achieves state-of-the-art performance on omni-modal benchmarks like Omni-Bench and WorldSense among open-source models. It also delivers highly competitive results across a wide range of modality-specific tasks, including text, image, and video understanding, as well as audio understanding and generation. Subjective evaluations confirm its ability to provide high-quality, low-latency audio-visual interactions.
The Meituan LongCat Team has open-sourced the model and provided a comprehensive overview of its architecture, training procedures, and data strategies to foster future research and development in the community. You can find the full technical report here.
