TLDR: TurboBias is a new, universal context-biasing framework for Automatic Speech Recognition (ASR) that significantly improves the recognition of specific key phrases. It works with all major ASR models (CTC, Transducers, AED) by using a GPU-accelerated phrase-boosting tree with a unique scoring method. This allows for high accuracy in both greedy and beam search decoding with minimal speed slowdown (2-5% RTFx), even with up to 20,000 key phrases. The method outperforms existing open-source solutions in accuracy and speed and is open-sourced as part of the NeMo toolkit.
Automatic Speech Recognition (ASR) systems have become highly accurate in general domains, but they often struggle with recognizing specific, rare, or domain-specific words and phrases, such as contact names, product titles, or technical terms. This challenge is addressed by ‘context-biasing’ methods, which aim to improve the accuracy of these key phrases.
Many existing context-biasing approaches come with significant drawbacks. Some ‘deep-fusion’ methods require extensive retraining of the ASR model or the development of additional context modules, which can be time-consuming and resource-intensive. ‘Shallow-fusion’ methods, which apply contextual information during the decoding stage without retraining, avoid this issue but often lead to a significant slowdown in the decoding process, especially when using ‘beam search’ – a technique that explores multiple recognition possibilities simultaneously to find the best one. This slowdown becomes particularly problematic with a large number of key phrases.
A new framework called TurboBias, powered by a GPU-accelerated Phrase-Boosting (GPU-PB) Tree, has been introduced to overcome these limitations. This innovative approach offers a universal solution for ASR context-biasing, supporting all major ASR model types: Connectionist Temporal Classification (CTC), Recurrent Neural Transducers (RNN-T), and Attention Encoder-Decoder (AED) models. The core of TurboBias lies in its GPU-accelerated phrase-boosting tree, which incorporates a modified scoring weight distribution. This design allows it to be used in shallow fusion mode for both ‘greedy’ (fast, single-path) and beam search decoding with minimal impact on speed, even when dealing with a vast number of key phrases, up to 20,000 items.
How TurboBias Works
The standard shallow fusion process involves increasing the probability of recognizing key phrases from a predefined context list. Traditional methods often use a prefix tree built with algorithms like Aho–Corasick. However, these can be slow or limited to CPU-only operations, especially when needing to query the entire vocabulary efficiently.
TurboBias addresses this by combining an efficient GPU tree architecture, inspired by the NGPU-LM (N-Gram Language Model) framework, with a phrase-boosting tree. A key innovation is its modified weight distribution. Instead of just assigning a reward at the final node of a phrase, TurboBias significantly increases scores for all tree transitions based on their depth. This logarithmic dependency on depth (c0 * β + ln(depth)) ensures that the boosting tree can be effectively traversed even in greedy search mode, where only one hypothesis is pursued at each step. This approach approximates the weight distribution of an n-gram language model for positive scores, making it effective for both greedy and beam search decoding.
The prefix tree, built from tokenized phrases in the context list, is converted into an efficient GPU-based structure. This structure enables rapid queries across the entire vocabulary using ‘index-select’ operations during inference. An additional custom Triton kernel further accelerates tree operations, allowing integration with CUDA Graphs for CTC and RNN-T decoding. This GPU-optimized structure is crucial for handling a large number of context phrases with minimal decoding speed overhead.
Adaptation and Performance
TurboBias is designed to be compatible with all major ASR models. For greedy decoding in CTC and RNN-T models, a two-stage token selection process is used, where tokens are reweighted by the boosting tree after an initial greedy selection. For AED models, which use an End-Of-Sentence (EOS) symbol, TurboBias employs heuristics to prevent EOS suppression, ensuring that the boosting scores don’t unfairly outweigh the EOS score, which could lead to decoding errors.
Experimental evaluations demonstrated the high efficiency of TurboBias. It significantly improved the accuracy of key phrase recognition (F-score) and also led to additional improvements in overall speech recognition accuracy (Word Error Rate, WER). For instance, in beam search decoding, TurboBias improved F-score by 17-23% and WER by 2-3% compared to baselines without context-biasing. Even in greedy decoding, F-score improvements of 8-10% were observed, which is crucial for streaming applications.
Crucially, TurboBias maintained excellent decoding speed. It introduced an average slowdown of only 2-5% in Real-Time Factor (RTFx) across all ASR models and datasets, even with context lists containing up to 1,000 phrases. This minimal overhead highlights the efficiency of its GPU implementation.
Also Read:
- New GPU Toolkit Boosts Speed and Accuracy in CTC Speech Decoding
- Unlocking Parallel Processing for Automatic Speech Recognition with Whisfusion
Comparison and Robustness
When compared to other open-source context-biasing solutions like Pyctcdecode, CTC-WS, and NGPU-LM, TurboBias generally outperformed them, especially in beam search, showing superior F-score and decoding speed. While some methods like CTC-WS showed good accuracy in greedy decoding, they often came with limitations such as requiring access to the full audio file (limiting streaming use) or being significantly slower due to CPU-based operations.
The framework also proved robust to variations in context list size. Even when the number of boosting phrases was increased from 200 to 20,000, the system’s runtime slowed by only about 5%, and recognition accuracy remained noticeably better than the baseline without phrase-boosting. This demonstrates its scalability for real-world applications with extensive vocabularies.
Furthermore, TurboBias can be used in conjunction with other language models, such as NGPU-LM, to achieve even greater reductions in WER, showcasing its complementarity with existing techniques.
In conclusion, TurboBias offers a powerful, universal, and highly efficient solution for ASR context-biasing. Its GPU-accelerated phrase-boosting tree with modified scoring provides significant accuracy improvements with minimal speed degradation, making it suitable for a wide range of ASR applications. The framework is openly available as part of the NeMo toolkit, fostering further development and adoption. You can find the full research paper here: TurboBias: Universal ASR Context-Biasing powered by GPU-accelerated Phrase-Boosting Tree.
