TLDR: A study introduces Whisper-LLaDA, an ASR system integrating a Whisper encoder with the LLaDA diffusion LLM. It significantly improves speech recognition accuracy through deliberation-based processing of existing transcripts, especially when audio-conditioned. As a standalone decoder, it offers faster inference with competitive, though sometimes slightly lower, accuracy compared to autoregressive baselines. The research highlights the critical role of audio-conditioned embeddings for effective diffusion-based ASR.
Automatic Speech Recognition (ASR) has seen significant advancements, largely due to sophisticated decoder models and autoregressive (AR) decoding methods that generate speech tokens sequentially. While effective, these AR approaches often come with high computational costs and reduced efficiency due to their sequential nature.
To address these limitations, non-autoregressive (NAR) Transformer decoders emerged, allowing for greater parallelization and improved decoding efficiency. However, NAR methods typically involve a slight trade-off, often resulting in a minor degradation in recognition accuracy compared to their AR counterparts.
A recent study introduces a promising new direction: integrating diffusion-based large language models (DLLMs) into ASR systems. Specifically, the research explores the use of LLaDA, a diffusion-based LLM, in two key ways: as an external module for refining existing transcripts (deliberation-based processing) and as a standalone decoder for ASR.
Understanding Diffusion-based LLMs and LLaDA
Diffusion-based LLMs operate on a generative principle involving a forward masking process and a learned reverse process. Imagine starting with a clean text sequence, where tokens are gradually replaced with a mask token. The reverse process then iteratively predicts these masked tokens, conditioned on the unmasked ones, to reconstruct the original coherent sequence. LLaDA implements this with a bidirectional attention-based Transformer, predicting masked positions in parallel.
Whisper-LLaDA: A New ASR Architecture
The proposed system, Whisper-LLaDA, combines the powerful Whisper-Large-v3 encoder with the LLaDA-8B-Instruct diffusion LLM. The Whisper encoder extracts speech representations, which are then processed through a window-level Query Transformer (Q-Former) and a projection layer to align them with the LLaDA model. Only specific components like LoRA, the Q-Former, and the projection layer are trained, making the system efficient.
Deliberation-based Processing: Refining Transcripts
One of the primary applications explored is using Whisper-LLaDA as an external deliberation module to refine transcripts generated by other ASR systems, such as Whisper-LLaMA. This involves taking an initial transcript, masking a proportion of its tokens, and then having Whisper-LLaDA reconstruct them, leveraging its bidirectional attention and denoising capabilities. The study investigated several masking strategies:
- Random Masking: Randomly selecting tokens to mask.
- Low-Confidence Masking: Masking tokens identified as having low confidence by Whisper-LLaDA.
- Semi-Autoregressive Deliberation: Dividing the transcript into sub-blocks and remasking them sequentially.
Crucially, the research found that audio-conditioned embeddings are vital for effective deliberation. A plain-text version of LLaDA, without acoustic features, failed to improve accuracy, highlighting the importance of integrating audio information. With audio conditioning, Whisper-LLaDA substantially reduced Word Error Rate (WER) compared to the Whisper-LLaMA baseline. For instance, on the LibriSpeech test-other split, the best cascade system achieved a 12.3% relative improvement over the baseline.
LLaDA as a Standalone ASR Decoder
The study also evaluated Whisper-LLaDA as a direct decoder for ASR, using both diffusion-based and semi-autoregressive decoding methods. Diffusion-based decoding involves iteratively predicting masked tokens over a set number of denoising steps. Semi-autoregressive decoding divides the response into sub-blocks, performing diffusion within each block and proceeding autoregressively across blocks.
While most configurations achieved faster inference times than the Whisper-LLaMA baseline, the recognition accuracy was slightly lower in some cases. However, certain settings, like the 4-block semi-autoregressive decoding with 32 steps per sub-block, showed competitive performance, particularly on the test-other dataset, surpassing LLaMA/Vicuna-based ASR baselines.
Also Read:
- Spiffy: A New Algorithm Boosts Diffusion LLM Speed Without Losing Quality
- Advancing Spoken Language Assessment with a Unified Multimodal AI Model
Key Takeaways and Future Directions
This empirical study demonstrates the significant potential of diffusion-based LLMs like LLaDA in ASR. It excels in deliberation-based processing, consistently improving recognition quality when conditioned on acoustic features. As a standalone decoder, it offers efficient inference. However, its accuracy in some direct decoding scenarios still lags behind extensively pre-trained autoregressive systems.
The findings suggest promising avenues for future research, including scaling Whisper-LLaDA with larger and more diverse training data beyond LibriSpeech, and exploring more advanced masking and remasking policies to further bridge the accuracy gap while maintaining efficiency. For more details, you can refer to the original research paper.
