TLDR: The Latent Speech-Text Transformer (LST) is a new model that improves speech-text understanding and generation by compressing long speech token sequences into shorter, information-dense “latent speech patches.” This reduces computational costs, enhances alignment between speech and text, and leads to better performance on benchmarks compared to traditional methods, especially when scaling up model size.
Large language models (LLMs) have revolutionized how we interact with text, demonstrating remarkable capabilities in understanding and generating human language. Inspired by this success, researchers have been developing similar models for speech, known as Generative Spoken Language Models (GSLMs). These models typically convert raw speech into discrete “speech tokens” and then train an auto-regressive language model on these tokens, mirroring how text LLMs process textual information.
However, a significant hurdle in the development of speech-based LLMs is the inherent difference in information density between speech and text. Representing the same semantic content often requires a disproportionately larger number of speech tokens compared to text tokens. This leads to several challenges: a substantial increase in computational costs during both model training and inference, and a potential hindrance to effectively aligning speech and text representations. Ultimately, this can result in much slower scaling laws for speech models compared to their text counterparts.
To tackle these challenges, a new research paper introduces an innovative solution: the Latent Speech-Text Transformer (LST). The core idea behind LST is to make the pre-training of speech-text models more data-efficient by dynamically and inexpensively aggregating speech tokens into “latent speech patches.” These patches serve as higher-level units that can either align with corresponding textual units to facilitate capability transfer or encapsulate common speech sequences, such as silences, to be more compute-efficient.
The LST architecture is inspired by the byte-latent transformer (BLT) and comprises three main components. First, a patch encoder dynamically groups sequences of speech tokens into these higher-level speech patches. Second, a global speech-transformer then auto-regressively models interleaved sequences of textual tokens and these newly formed speech patches. Finally, a lightweight transformer decoder maps these patches back into speech tokens of dynamic sizes. By operating on these information-dense speech patches instead of individual, granular speech tokens, the model significantly reduces the computational expense, particularly within the global transformer.
The researchers explored various strategies for creating these speech patches. “Static Patching” involves segmenting speech into non-overlapping, fixed-length patches. A more advanced method, “Alignment Patching,” leverages forced alignment timestamps between speech frames and textual units (like words or subword tokens). This allows patches to correspond directly to semantic units, improving synchronization between speech and text. A particularly effective strategy introduced is “Curriculum Patching,” which begins training with alignment-based patching to harness its benefits and gradually transitions to static patching. This approach ensures robustness during inference without relying on auxiliary alignment models.
Experiments conducted on popular benchmarks, including HellaSwag, StoryCloze, and TopicStoryCloze, demonstrated that LST models consistently outperform traditional approaches. For example, on HellaSwag story completion, LST achieved a 6.5% absolute gain in speech accuracy under compute-controlled training and a 5.3% gain under data-controlled training, while also improving text performance. These results highlight a more effective representational alignment between speech and text and indicate steeper scaling laws for speech-text models.
Furthermore, the LST framework shows promising scalability. When scaled from 1 billion to 7 billion parameters, LST models continued to outperform baselines, exhibiting a steeper growth curve over training iterations. This suggests that LST can utilize larger model capacities more efficiently, leading to even greater advantages with extended training.
The research underscores that the significant mismatch in information densities between speech and text tokens has been a primary factor hindering effective speech-text alignment. By introducing speech patches, LST effectively levels this information density, making speech and text easier to align and process. This innovation not only saves considerable training and inference compute but also significantly improves speech-text representational alignment, paving the way for more efficient and capable multimodal AI systems.
Also Read:
- TokenChain: Unlocking Efficient Speech AI with Discrete Semantic Tokens
- Boosting Transformer Efficiency with Compressed Convolutional Attention
For more in-depth details, you can refer to the full research paper: Latent Speech-Text Transformer.
