TLDR: AUDIOMARATHON is a new benchmark for Large Audio Language Models (LALMs) designed to evaluate their ability to understand and efficiently process long-form audio (90-300 seconds). It covers diverse audio types (speech, sound, music) and complex reasoning tasks. Initial evaluations show current LALMs struggle with long-context understanding and efficiency, highlighting the need for better temporal reasoning and memory-efficient architectures, and demonstrating the effectiveness of tailored token pruning strategies.
Large Audio Language Models (LALMs) are powerful tools for understanding spoken language, environmental sounds, and music. However, they face significant hurdles when dealing with long audio inputs, such as extended meetings or podcasts. The main issues are the high computational and memory costs associated with processing long sequences, and the difficulty in accurately modeling long-range temporal dependencies within the audio.
Current audio benchmarks primarily use short audio clips, which don’t accurately reflect the challenges LALMs encounter in real-world, long-context scenarios. To bridge this gap, researchers have introduced AUDIOMARATHON, a new benchmark designed to rigorously evaluate both the understanding capabilities and inference efficiency of LALMs on long-form audio.
AUDIOMARATHON is built on three core principles. First, it uses long-context audio inputs, ranging from 90 to 300 seconds, which translate to 2,250 to 7,500 audio tokens. This duration is crucial for simulating realistic usage. Second, it offers full domain coverage, including speech, environmental sounds, and music, ensuring a diverse evaluation. Third, it incorporates complex reasoning tasks that require models to perform multi-hop inference, meaning they need to connect multiple pieces of information across extended timeframes to arrive at an answer.
The benchmark includes a variety of tasks categorized into Speech Context Understanding (like Automatic Speech Recognition, Speech Content Reasoning, and Speech Entity Recognition), Audio Scene Understanding (such as Music Classifier, Audio Scene Classifier, and Sound Event Detection), and Voice Characteristic Identification (including Emotion Recognition, Speech Detection, Speaker Age Recognition, and Speaker Gender Recognition).
Initial evaluations of state-of-the-art LALMs on AUDIOMARATHON revealed a clear drop in performance as audio length increased. This highlights the existing limitations of these models in handling extended audio. The study also explored acceleration techniques like token pruning and KV cache eviction, analyzing their trade-offs. The results indicate substantial performance differences among current LALMs and emphasize the need for better temporal reasoning and more memory-efficient architectures.
For instance, tasks like long-form Speech Entity Recognition and Speaker-Age Recognition proved to be particularly challenging for the models. Even the best-performing models struggled to achieve high scores in these areas, indicating a significant gap compared to human performance. This suggests that current LALMs still have a way to go in tasks requiring detailed entity tracking and complex temporal reasoning over long audio segments.
The research also delved into efficiency optimizations. Processing a five-minute audio input can generate thousands of tokens, leading to massive memory requirements and slow inference. Token pruning, which reduces the number of tokens during the initial processing stage, and KV-cache eviction, which manages memory during the decoding stage, were evaluated. While these methods offer significant speedup and memory savings, careful implementation is crucial. For example, aggressive pruning can sometimes remove important phonetic cues, especially in tasks like Automatic Speech Recognition, leading to degraded accuracy.
A key finding was that “Frame” pruning, a time-aligned token pruning strategy, generally performed better for speech-centric tasks by preserving rare or short-lived acoustic events that other methods might discard. This underscores the importance of task-aware token compression strategies tailored to the unique characteristics of audio data.
Also Read:
- Beyond Text: AURA Score for Better Audio Question Answering Evaluation
- Bridging the Gap Between Speech and Text Models with Latent Speech Patches
The AUDIOMARATHON benchmark is expected to encourage the audio and multimodal research community to develop more advanced audio understanding models that can efficiently tackle complex, long-form audio tasks. This work provides a crucial foundation for building robust and efficient LALMs capable of truly comprehending the continuous and rich tapestry of the auditory world. You can find the full research paper here.
