TLDR: This paper introduces Router-Gated Cross-Modal Feature Fusion, a novel audio-visual speech recognition (AVSR) framework. It adaptively reweights audio and visual features based on token-level acoustic corruption scores, using a router to identify unreliable audio. Through gated cross-attention, the system reinforces visual cues when audio quality is poor, leading to significant word error rate reductions (16.51–42.67%) on LRS3 compared to AV-HuBERT, and demonstrating improved robustness in noisy conditions.
Speech recognition systems that combine both audio and visual information, known as Audio-Visual Speech Recognition (AVSR), hold great promise for understanding speech, especially in noisy environments where audio alone might be unclear. Imagine trying to understand someone speaking in a loud cafe; seeing their lips move can significantly help. However, current AVSR systems often struggle to effectively use visual cues when the audio quality drops, remaining too reliant on the audio stream.
A new research paper titled “Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion” introduces a novel approach to tackle this challenge. The core problem with existing systems is their inability to dynamically adjust how much they rely on audio versus visual information based on how noisy the audio is. Many models are “audio-centric,” meaning they prioritize sound even when it’s heavily corrupted, leading to limited performance gains in truly challenging conditions.
The researchers propose a framework called Router-Gated Cross-Modal Feature Fusion. This innovative system is designed to adaptively reweight audio and visual features. It does this by assessing the “acoustic corruption” of the audio at a very fine-grained, token-level. Think of a “token” as a small segment of speech. If a token of audio is deemed unreliable due to noise, the system can then reduce its dependence on that audio segment and instead reinforce the visual cues, like lip movements.
How the Router Works
At the heart of this system is an “audio-visual feature fusion-based router.” This router acts like a smart sensor, computing the similarity between audio and visual embeddings (representations) without needing explicit labels about noise. A low similarity score between the audio and what the visual information predicts the audio should be, signals that the audio is likely corrupted. When this happens, the router tells the system to pay more attention to the visual information.
Gated Cross-Attention for Dynamic Fusion
This increased reliance on visual cues is achieved through a mechanism called “gated cross-attention” within each layer of the decoder. The router’s reliability score acts as a “gate,” dynamically controlling how much visual information is integrated. This allows the model to “pivot” towards the visual modality precisely when the audio quality deteriorates, mimicking how humans might instinctively focus on lip-reading when it’s hard to hear.
Unlike previous methods that might make broad decisions about audio reliability over longer segments, this new approach operates at a fine-grained, token level and adapts across different layers of the network. This makes the fusion process highly context-sensitive. Crucially, it can be integrated as a “plug-in” module into existing Transformer-based decoders, such as AV-HuBERT and Whisper-Flamingo, without requiring extensive retraining of the base model on specific noise types.
Also Read:
- Enhancing ASR Reliability: A Framework for Internal Consistency Across Semantic and Structural Levels
- Efficient Lip-Reading: New Lightweight Models Make Visual Speech Recognition Practical
Experimental Results
The effectiveness of this framework was demonstrated through experiments on the LRS3 dataset. The proposed method achieved a significant reduction in word error rate (WER) ranging from 16.51% to 42.67% compared to AV-HuBERT, a leading AVSR model. This improvement was observed under various real-world acoustic noise conditions, including babble, music, and natural noise, at different signal-to-noise ratios (SNRs). The system also showed strong generalization capabilities, performing well on the unseen LRS2 dataset and proving compatible with the Whisper-Flamingo architecture, further highlighting its versatility.
Ablation studies, where components of the system are removed to test their individual contributions, confirmed that both the router and the gating mechanism are vital for the improved robustness. The research highlights that this dynamic and noise-aware fusion mechanism is a significant step forward in making AVSR systems more reliable and effective in challenging, noisy environments. For more technical details, you can read the full paper here.
