TLDR: FunAudio-ASR is a large-scale, LLM-based automatic speech recognition system that combines extensive data, model capacity, LLM integration, and reinforcement learning to achieve state-of-the-art performance. It is specifically optimized for practical deployment, excelling in streaming, noise robustness, code-switching, and hotword customization, outperforming other systems on real-world industry evaluation sets while mitigating LLM hallucination.
Automatic Speech Recognition (ASR) has seen remarkable progress recently, driven by three main factors: using vast amounts of data, increasing the size of models, and integrating deeply with large language models (LLMs). While LLMs have brought significant improvements, they also introduce a challenge: hallucination, where the system generates text not present in the audio, which can negatively impact user experience in real-world applications.
Addressing this, the FunAudio Team from Tongyi Lab, Alibaba Group, presents FunAudio-ASR, a large-scale, LLM-based ASR system. This system brings together massive datasets, substantial model capacity, LLM integration, and reinforcement learning to achieve top-tier performance across various complex speech recognition scenarios. What sets FunAudio-ASR apart is its specific optimization for practical deployment, with enhancements in streaming capabilities, robustness to noise, handling code-switching, customizable hotword recognition, and meeting other real-world application demands.
The Core of FunAudio-ASR
FunAudio-ASR is built on the synergistic benefits of data scaling, model size scaling, and innovative LLM integration. It aims for state-of-the-art speech recognition accuracy across diverse linguistic and acoustic environments. Beyond benchmark performance, it is meticulously engineered for real-world use, focusing on usability, reliability, and user experience. This includes an efficient streaming ASR architecture for real-time processing, enhanced noise robustness through a multi-stage approach, advanced code-switching capabilities for seamless transitions between languages like Chinese and English, and customizable hotword recognition for domain-specific terms.
Model Architecture and Data Foundation
The FunAudio-ASR model consists of four key components: an audio encoder to extract speech representations, an audio adaptor to connect the encoder output with the LLM, a CTC decoder for initial recognition hypotheses (used for hotword customization), and an LLM-based decoder to produce the final output. To cater to different computational needs, two models are proposed: FunAudio-ASR (0.7B encoder, 7B LLM-based decoder for high accuracy) and FunAudio-ASR-nano (0.2B encoder, 0.6B LLM-based decoder for efficiency).
The system is trained on an immense scale of data. Pre-training involves tens of millions of hours of audio, including both unlabeled and labeled audio-text data, primarily in Chinese and English. Supervised fine-tuning (SFT) utilizes millions of hours of diverse data, including human-transcribed, pseudo-labeled, environmental noise, and synthetic data.
Advanced Training Paradigms
The training of FunAudio-ASR is a multi-stage process. The audio encoder undergoes self-supervised pre-training using the Best-RQ framework, notably initialized with weights from a pre-trained text LLM (Qwen3 model) to leverage existing linguistic knowledge. This is followed by supervised pre-training within an attention-based encoder-decoder framework. Supervised fine-tuning then proceeds through several stages, aligning audio representations with the LLM, training the encoder and adaptor, adapting LLM parameters with Low-Rank Adaptation (LoRA), and finally, full-parameter fine-tuning with high-quality data.
A crucial aspect is Contextual Supervised Fine-tuning, which expands the model’s contextual modeling capability by training on long-duration and contextual data. This involves extracting keywords and synthesizing relevant and irrelevant contextual content using the Qwen3-32B model.
Reinforcement Learning for Refinement
FunAudio-ASR incorporates FunRL, an efficient reinforcement learning (RL) framework designed for large audio-language models. This framework uses a GRPO-based RL algorithm with a set of custom value functions to enhance ASR performance and user experience. These functions optimize for ASR accuracy, keyword accuracy and recall, noise robustness, hallucination suppression, and language matching. The RL training data is carefully constructed to include ‘hardcase’ samples (where FunAudio-ASR struggles compared to other systems), long-duration samples, hallucination-related samples, keyword/hotword samples, and regular ASR data to ensure comprehensive improvement.
Production-Oriented Optimizations in Detail
- Streaming Ability: The model is fine-tuned with simulated streaming-style training data to reduce the mismatch between training and real-time inference.
- Noise Robust Training: A large-scale noisy data augmentation strategy, combining low-noise speech with noise samples, significantly improves performance in challenging acoustic conditions.
- Multilingual ASR: An additional FunAudio-ASR-ML model supports Mandarin Chinese, English, Vietnamese, Thai, and Indonesian, with balanced data distribution during training.
- Code-switching: Synthesized Chinese-English code-switched training data, generated using Qwen3 and a Text-to-Speech model, enhances recognition in mixed-language speech.
- Hotword Customization: A Retrieval-Augmented Generation (RAG)-based mechanism allows users to define domain-specific hotwords, improving recognition accuracy for technical terms or names.
- Hallucination Mitigation: By introducing zero-padding into audio signals before adding noise, the model learns to recognize pure-noise segments, reducing spurious transcriptions.
Evaluation and Real-World Performance
FunAudio-ASR and FunAudio-ASR-ML were rigorously evaluated on both open-source benchmarks and real-world industry evaluation sets. While many LLM-based ASR systems perform well on open-source data, they often fall short on industry-specific datasets. FunAudio-ASR, however, achieves state-of-the-art performance on both, demonstrating its effectiveness and robustness in practical settings. Notably, FunAudio-ASR-nano, with only 0.8B parameters, also outperforms open-source models and performs comparably to larger systems like Seed-ASR.
The evaluations confirmed significant improvements in streaming ASR, noise robustness (especially in challenging environments like dinner and supermarkets), code-switching, and hotword customization. Reinforcement learning played a crucial role, bringing substantial relative improvements in both offline and streaming conditions, particularly in noisy and complex environments, and enhancing hotword integration.
Also Read:
- Unifying Speech Understanding: How FuseCodec Integrates Meaning and Context into Digital Speech
- Enhancing Air Traffic Control Communications with Specialized AI Speech Recognition
Future Directions
Despite its strong performance, FunAudio-ASR has limitations. It is primarily optimized for Chinese and English, with limited support for other languages. Its effective context window is constrained without an external voice activity detection (VAD) module, making it challenging to handle very long recordings. Additionally, the current release does not support far-field or multi-channel audio. The FunAudio Team plans to address these limitations in future work.
In conclusion, FunAudio-ASR represents a significant leap in LLM-based ASR, effectively bridging the gap between academic research and commercial readiness. Its production-oriented design and optimizations make it a high-performance, deployable solution for real-world speech recognition challenges. You can read the full technical report here.
Authors: Keyu An, Yanni Chen, Chong Deng, Changfeng Gao, Zhifu Gao, Bo Gong, Xiangang Li, Yabin Li, Xiang Lv, Yunjie Ji, Yiheng Jiang, Bin Ma, Haoneng Luo, Chongjia Ni, Zexu Pan, Yiping Peng, Zhendong Peng, Peiyao Wang, Hao Wang, Wen Wang, Wupeng Wang, Biao Tian, Zhentao Tan, Nan Yang, Bin Yuan, Jieping Ye, Jixing Yu, Qinglin Zhang, Kun Zou, Han Zhao, Shengkui Zhao, Jingren Zhou.
