TLDR: AudioCodecBench is a new benchmark that provides a comprehensive and fair evaluation framework for audio codecs, which are crucial for Multimodal Large Language Models (MLLMs). The paper redefines ‘acoustic’ and ‘semantic’ audio features and evaluates codecs across four dimensions: audio reconstruction, codebook ID stability, LM perplexity, and performance on various downstream tasks (speech, music, sound). Key findings indicate that semantic-rich tokens are easier for LMs to model and perform better in high-level tasks, while purely acoustic codecs excel in reconstruction but are more sensitive to noise. The benchmark aims to guide the development of more effective audio-semantic models.
In the rapidly evolving landscape of Multimodal Large Language Models (MLLMs), the way audio is processed and understood by these models is becoming increasingly crucial. Unlike text, which primarily conveys semantic meaning, audio carries a dual burden: it must capture both the overarching semantic content (what is being said or the emotion of music) and the intricate, fine-grained acoustic details (the specific sound of a voice or instrument). This unique challenge has led to a focus on audio tokenization, a process that converts continuous audio waveforms into discrete units that MLLMs can process.
However, existing research has struggled with clear definitions of what constitutes a “semantic token” versus an “acoustic token.” Furthermore, the evaluation of different audio codecs – the systems that perform this tokenization – has often been limited to specific domains or tasks, making it difficult to conduct fair and comprehensive comparisons. To address these significant gaps, a new research paper introduces AudioCodecBench, a comprehensive benchmark designed to systematically evaluate audio codecs.
Redefining Audio Features
The paper begins by providing clear and suitable definitions for different types of audio features, emphasizing that a semantic feature must be strictly describable by text. Based on this, it categorizes audio features into four types:
- Acoustic features: These are discrete features that cannot be described by text. They come from codecs optimized for reconstructing acoustic details like environmental noise or vocal vibrations.
- Semantic features: These are discrete features extracted from self-supervised learning (SSL) models, strictly defined by text. They capture high-level information such as transcribed speech, music emotion, or the presence of a human voice.
- Semantic-Acoustic fused features: These features combine both text-describable semantics and text-indescribable acoustic information. An example would be a speaker’s voice, which contains both the textual content and unique acoustic characteristics.
- Semantic-Acoustic decoupled features: These features separate text-describable semantics and text-indescribable acoustics into independent token streams. For instance, an audio clip of “Hello” might yield one stream for the text “Hello” and another for the speaker’s unique acoustic signature.
A Multi-Dimensional Evaluation Framework
AudioCodecBench introduces a systematic evaluation framework that assesses codecs across four key dimensions:
- Audio Reconstruction: This evaluates how accurately a codec can reconstruct the original audio waveform after encoding and decoding. Metrics like Perceptual Evaluation of Speech Quality (PESQ) and Short-Time Objective Intelligibility (STOI) measure acoustic fidelity, while Word Error Rate (WER) and Character Error Rate (CER) assess semantic preservation.
- Codebook ID Stability (ID Sensitivity): This measures the robustness of discrete tokens under noisy conditions. It includes multi-round reconstruction (re-encoding reconstructed audio multiple times) and temporal shift stability (introducing slight time shifts). Higher stability indicates lower ID sensitivity.
- Decoder-Only Transformer Perplexity: This evaluates how well a sequence of discrete audio tokens can be modeled by a Large Language Model (LM). Lower perplexity suggests that the token sequence is more predictable and potentially contains richer semantics, making it easier for LMs to learn.
- Performance on Downstream Probe Tasks: This assesses the generalization capabilities of discrete tokens across various real-world applications. These tasks span three audio domains: speech (e.g., Automatic Speech Recognition, Emotion Detection), music (e.g., Genre Classification, Key Detection, Music Tagging), and sound (e.g., Vocal Sound Classification, Environmental Sound Classification).
Key Findings and Insights
The experiments conducted using AudioCodecBench reveal several important insights:
- Reconstruction: Codecs primarily focused on acoustic reconstruction (like DAC and EnCodec) generally achieve the highest fidelity. However, codecs that integrate semantic information often show slightly suboptimal reconstruction, suggesting a trade-off. Music, with its complex harmonic structures, is significantly harder to reconstruct than speech.
- ID Sensitivity: Acoustic codecs tend to be more sensitive to temporal changes and noise, as they try to capture every detail. In contrast, codecs that integrate semantics demonstrate greater robustness to these perturbations, focusing on stable content features.
- Perplexity: Semantic tokens are consistently easier for LMs to model, resulting in lower perplexity scores. This highlights the benefit of semantic content for language model adaptability. Music datasets generally yield higher perplexity than speech, reflecting their greater complexity.
- Downstream Probe Tasks: Codecs that integrate semantics generally outperform purely acoustic codecs across a wide range of tasks. For instance, they show advantages in music tasks involving high-level structures (like genre and key detection) and perform better in speech tasks like ASR and Vocal Sound Classification. Self-supervised learning (SSL) models, which are designed to capture rich representations, achieve the best performance overall in these tasks.
The research also explores the correlation between perplexity and other metrics. It found that perplexity is positively correlated with semantic-rich probe tasks, confirming that tokens with strong semantic content are more amenable to LM modeling. Conversely, it shows a negative correlation with objective acoustic reconstruction metrics, implying that an excessive focus on acoustic details might make LM modeling more challenging.
Also Read:
- Enhancing Omni-Modal Language Models: A New Framework to Combat Hallucinations
- FLM-Audio: A Breakthrough in Real-Time Conversational AI
A New Direction for Audio Representation
AudioCodecBench provides a much-needed, comprehensive, and fair evaluation framework for audio codecs. By redefining acoustic and semantic features and evaluating performance across multiple dimensions and audio domains, this benchmark offers valuable insights into the strengths and weaknesses of different audio tokenization methods. The findings not only highlight the potential applications of various audio representations in MLLMs but also point towards a promising new research direction: developing better audio-semantic models by more effectively aligning audio with text modalities. The paper and its associated code and dataset are available for further exploration at AudioCodecBench: A Comprehensive Benchmark for Audio Codec Evaluation.
