TLDR: A new method called “Whispering Context” improves Automatic Speech Recognition (ASR) by distilling linguistic knowledge from LLaMA models into Whisper. This approach uses token-level and representation-level distillation, along with an extended context window, to significantly enhance accuracy in long speech transcripts, particularly for punctuation, capitalization, and Named Entity Recognition (NER), including challenging “long-tail” entities. The resulting ASR system is more robust and produces highly formatted, context-aware outputs.
Automatic Speech Recognition (ASR) systems are essential for converting spoken language into text, but they often face significant challenges when dealing with long audio recordings. Maintaining accuracy in terms of grammar (syntax) and meaning (semantics) in these lengthy transcripts is particularly difficult, impacting crucial downstream tasks like identifying named entities (NER), correctly capitalizing words, and applying proper punctuation. Existing ASR models, even powerful ones like Whisper, struggle with these aspects, especially text normalization and entity formatting, despite being trained on vast amounts of audio data.
A new research paper titled “Whispering Context: Distilling Syntax and Semantics for Long Speech Transcripts” introduces a novel approach to overcome these limitations. The core idea is to enhance Whisper, a widely used ASR model, by transferring the rich linguistic knowledge from Large Language Models (LLMs) like LLaMA. This process, known as knowledge distillation, aims to teach Whisper how to produce more accurate and fully formatted transcriptions, similar to what a human would write.
How the Enhancement Works
The researchers propose two main strategies for this knowledge transfer. First, they use a technique called token-level distillation, which involves aligning the output probabilities (logits) of LLaMA and Whisper. To handle the differences between text and speech modalities, an advanced mathematical method called optimal transport is employed. This ensures that the models’ internal representations are aligned, allowing Whisper to learn from LLaMA’s understanding of individual words and their contexts.
Second, they minimize the difference in the “sentence embeddings” between Whisper and LLaMA. Sentence embeddings are numerical representations that capture the overall meaning and structure of a sentence. By making Whisper’s sentence embeddings similar to LLaMA’s, the system learns to blend both syntactic (grammatical) and semantic (meaning-related) understanding, leading to more coherent and accurate transcriptions.
The Power of Extended Context
A significant innovation in this research is the use of an “extended context window” during the distillation process. Instead of only considering the immediate 30-second audio chunk that Whisper processes, the model is given a much broader context from the entire transcript, including text that comes before and after the current chunk. This means Whisper can leverage more semantic information from the surrounding text, which proved especially effective for recognizing “long-tail entities” – rare or complex names and terms that are difficult to identify without a wider understanding of the conversation.
The study utilized the Spoken Wikipedia dataset, known for its long-form audio and diverse entity types, to evaluate the approach. The LLaMA models (1B, 3B, and 8B parameters) were fine-tuned on formatted transcripts, and the medium-sized Whisper model was fine-tuned on audio-transcript pairs, with its vocabulary augmented to include specific entity tags.
Impressive Results
The results of this distillation process were remarkable. The Word Error Rate (WER), a common metric for ASR accuracy, significantly improved. The baseline Whisper-Medium had a WER of 0.38, which improved to 0.26 after fine-tuning and further dropped to an impressive 0.20 with the LLaMA-distilled approach. This indicates a substantial reduction in transcription errors.
Beyond just word accuracy, the improvements in punctuation and Named Entity Recognition (NER) were particularly noteworthy. For punctuation, the distilled model showed major gains in correctly placing commas, periods, semicolons, exclamation marks, and question marks. For instance, semicolons, which were completely missed by the fine-tuned Whisper (F1 score of 0.00), achieved an F1 score of 0.61 with distillation, highlighting LLaMA’s contribution to syntactic understanding. Exclamation marks went from 0.00 to a perfect 1.00 F1 score.
NER performance also saw significant boosts across all entity types. Numerical entities (like dates, quantities, money) showed dramatic improvements, with near-zero Character Error Rate (CER), meaning they were almost perfectly formatted. Textual entities, such as locations and events, also benefited greatly, demonstrating the model’s enhanced semantic understanding. The ability to correctly transcribe and format complex or rare entities, often referred to as “long-tail entities,” was a key success, as the model could leverage broader context to resolve ambiguities.
Furthermore, the LLaMA-distilled Whisper proved to be more robust to noise in NER annotations, maintaining strong performance even with up to 25% corruption in the training data, unlike the standard Whisper model which quickly degraded. This suggests that the distilled knowledge provides a more resilient system.
Also Read:
- Enhancing Speech Recognition in Noisy Environments with HuBERT-VIC
- Advancing Automated Speaking Assessment with Multimodal AI and Speech-First Learning
Looking Ahead
This research marks a significant step forward in ASR technology. By effectively transferring the linguistic intelligence of Large Language Models into ASR systems, the work demonstrates a powerful method for generating high-quality, fully formatted speech transcriptions that are both human-readable and machine-usable. It paves the way for more robust and context-aware ASR systems, especially for long-form speech. You can read the full research paper here.
