TLDR: A new method called SALAD (Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation) addresses the performance difference between text-based and speech-adapted Large Language Models. It tackles two key issues: the forgetting of text capabilities and cross-modal misalignment. By combining knowledge distillation with targeted synthetic data, SALAD significantly reduces the amount of speech data needed for training while achieving competitive performance in language understanding and preserving the original text capabilities of the LLM.
Large Language Models (LLMs) have transformed how we interact with artificial intelligence, showcasing remarkable abilities in understanding and generating text. However, extending these powerful models to understand spoken language has presented a significant challenge. Researchers have observed a consistent performance drop when LLMs process speech inputs compared to when they handle equivalent text. This phenomenon is termed the ‘text–speech understanding gap’. A new research paper, “Closing the Gap Between Text and Speech Understanding in LLMs”, introduces an innovative approach to tackle this problem.
The current methods for adapting LLMs to speech often fall short. Some rely on generating vast amounts of synthetic speech from text, which is both expensive and heavily dependent on the quality of the synthetic data. Others use massive, proprietary speech datasets, making their findings difficult to reproduce or build upon. This highlights a critical need for more data-efficient solutions to bridge the text–speech understanding gap.
Understanding the Core Problem
The authors of this paper delve deeper into the reasons behind this performance disparity, identifying two primary factors: (i) the forgetting of text capabilities during the adaptation process to speech, and (ii) a cross-modal misalignment, meaning the model struggles to consistently interpret semantically equivalent speech and text inputs. Forgetting refers to the loss of the original text-based knowledge that the LLM acquired during its initial training, a common issue when models are fine-tuned on new domains. Cross-modal misalignment occurs when the model produces different outputs for speech and text inputs that convey the same meaning.
Introducing SALAD: A Sample-Efficient Solution
Based on their analysis, the researchers introduce a novel method called SALAD, which stands for Sample-efficient Alignment with Learning through Active selection and cross-modal Distillation. SALAD is designed to address both forgetting and cross-modal misalignment efficiently. It combines two key strategies: cross-modal distillation and targeted synthetic data generation.
Cross-modal distillation involves using the original text-based LLM as a ‘teacher’ to guide the speech-adapted LLM. This helps the speech model learn to produce outputs consistent with its text-based counterpart, thereby mitigating forgetting and improving alignment. The second strategy, active selection, is a data-efficient way to introduce synthetic data. Instead of synthesizing massive, broad-domain speech corpora, SALAD actively identifies specific areas where the model struggles with misalignment and then strategically generates a small, targeted amount of synthetic speech data for those areas. This ensures that the synthetic data is used precisely where it’s most needed, expanding the model’s domain coverage without excessive cost.
How SALAD Works in Practice
SALAD operates in two stages. Stage I focuses on ‘Distillation on Natural Speech’, where the model is trained on existing natural speech data using the cross-modal distillation objective. This stage leverages the strong scaling properties of distillation to achieve robust alignment and reduce forgetting. Once the alignment gains from natural speech data begin to plateau, Stage II, ‘Active Selection for Domain Expansion’, kicks in. In this stage, the model’s own misalignment signals guide the selection of text samples from a broad-domain text corpus. These selected samples are then synthesized into speech, creating a small but highly relevant dataset to address the remaining misalignment. This targeted approach ensures that the model learns from diverse domains that were underrepresented in the initial natural speech data.
Also Read:
- Decoding the Modality Gap in Speech Language Models
- Adapting Knowledge Distillation for Efficient Large Language Models
Impressive Results with Less Data
The SALAD method was applied to 3B and 7B LLMs and benchmarked against several strong open-weight models across a range of broad-domain tasks, including knowledge, language understanding, and reasoning. The results were highly competitive, with SALAD achieving performance comparable to the strongest existing models. Crucially, SALAD accomplished this while training on significantly less speech data – over an order of magnitude less – from public corpora. Furthermore, SALAD demonstrated superior preservation of the original text capabilities of the LLM backbone, unlike many other speech-adapted models that exhibit substantial forgetting.
This research highlights that a thoughtful combination of learning objectives and data selection strategies can dramatically reduce the reliance on expensive, large-scale synthetic data or proprietary speech resources. SALAD paves the way for more data-efficient and reproducible methods to close the text–speech understanding gap, bringing us closer to truly natural spoken interaction with AI systems.
