TLDR: A new method called Adaptive Knowledge Distillation (aKD) significantly improves Device-Directed Speech Detection (DDSD) accuracy for voice assistants on smart devices. This technique transfers knowledge from large, pre-trained ASR models to smaller, on-device models by simultaneously training the teacher’s task-specific adapters and the student model. By combining embedding, pseudo-labeling, and novel attention regularization losses, aKD achieves substantial reductions in Equal Error Rate (EER) for both keyword and keyword-free invocations, making voice assistants more reliable and efficient for on-device deployment.
Smart devices like phones, wearables, and smart speakers have become essential in our daily lives, largely due to voice assistants. Users interact with these devices through voice commands, touch, or keyword-free follow-up modes. A crucial technology behind this seamless interaction is Device-Directed Speech Detection (DDSD). DDSD is a classification task that distinguishes between a user’s direct queries to a voice assistant and other background speech or side conversations. This is vital for preventing unintended activations, which can happen if speech accidentally resembles a wake-word or if a button is pressed by mistake.
Improving the accuracy of DDSD is a significant challenge, especially when considering the limited computing resources available on smart devices. Large, highly accurate models are often too big for on-device deployment. To address this, researchers from Apple and Meta have proposed a novel approach called Adaptive Knowledge Distillation (aKD).
Knowledge Distillation (KD) is a technique where a smaller, more efficient ‘student’ model learns from a larger, more powerful ‘teacher’ model. The motivation here is twofold: the teacher model is trained on a much larger dataset and is more robust, while the student model needs to be small enough for on-device use. While KD has been explored in various fields like computer vision and general speech processing, its application to DDSD has been less investigated.
Adaptive Knowledge Distillation (aKD) Explained
The conventional KD process typically involves two steps: first, fine-tuning the teacher model for the specific task, and then distilling that knowledge to the student. This can be resource-intensive and often leaves a significant ‘knowledge gap’ between the already trained teacher and the untrained student at the start of distillation.
Adaptive Knowledge Distillation (aKD) streamlines this process by training the teacher’s task-specific adapters and the student model simultaneously. This means that as the teacher’s adapters are optimized for the DDSD task, the student model is learning from it in real-time. This dynamic alignment helps to minimize the knowledge gap from the very beginning of the training process, leading to more effective knowledge transfer.
The Distillation Losses
The aKD method integrates several types of distillation losses to ensure comprehensive knowledge transfer:
-
Embedding Distillation (ED): This loss encourages the student model’s acoustic encoder to mimic the robust, general representations learned by the teacher’s ASR (Automatic Speech Recognition) encoder. Since the teacher’s ASR encoder is frozen and not fine-tuned for DDSD, it provides a broad, beneficial representation.
-
Pseudo Labeling (PL): Here, the student model learns to mimic the DDSD predictions of the teacher model. Instead of using the ground truth labels, the student uses the teacher’s output probabilities as targets, helping it to emulate the teacher’s decision-making process.
-
Attention Regularization (AR): This is a novel component that addresses the temporal context, which is crucial for keyword detection. Not all parts of an audio input are equally important. AR aligns the global attention weights between the student and teacher models, ensuring the student understands which timeframes are most salient, effectively transferring temporal context.
These individual losses are combined into a single objective function, with specific weights determining their contribution to the overall training.
Model Architecture and Performance
The DDSD model takes audio features as input and classifies them as device-directed or not. The researchers used a unified acoustic detector architecture, employing both Transformer and Conformer-based acoustic encoders for the student models. The teacher model was a large Conformer-based ASR model with 79 million parameters, while the student models were significantly smaller, around 5 million parameters, making them suitable for on-device deployment.
Experiments demonstrated that aKD consistently outperforms conventional KD across both Transformer and Conformer student architectures and for all invocation types (keyword-based like ‘Hey Agent’ and ‘Agent’, and keyword-free follow-up conversations). The accuracy improvements were substantial: aKD achieved an average relative gain of 26% for keyword-based invocations and 19% for keyword-free invocations in terms of Equal Error Rate (EER).
Notably, the Conformer architecture, which had not been widely tested for DDSD before, showed superior performance compared to the Transformer-based model, aligning with its known advantages in the acoustic domain. The combination of all proposed distillation losses yielded the best or highly competitive performance, highlighting their complementary nature.
Also Read:
- EmoSteer-TTS: Precise Emotion Control in Synthesized Speech Without Retraining
- Audio AI’s Covert Weakness: How Benign Sounds Can Unleash Harmful Content
Conclusion
This research introduces Adaptive Knowledge Distillation as a highly effective method for improving device-directed speech detection accuracy on resource-constrained devices. By unifying various distillation losses and enabling simultaneous training of teacher adapters and the student model, aKD efficiently transfers knowledge from large, robust ASR models to smaller, deployable DDSD models. This approach significantly reduces the Equal Error Rate across different invocation types and model architectures, enhancing the user experience of voice-activated devices while maintaining on-device deployment feasibility. For more details, you can refer to the full research paper: Adaptive Knowledge Distillation for Device-Directed Speech Detection.
