TLDR: OWL is a new AI framework that significantly improves audio large language models’ ability to understand and reason about the 3D spatial location of sounds. It achieves this through a geometry-aware audio encoder (SAGE) trained with visual depth cues and a “chain-of-thought” reasoning process. A new dataset, BiDepth, was created to support its training and evaluation, enabling more accurate direction and distance estimation, and complex spatial reasoning compared to previous models.
Understanding where sounds come from in a 3D space is a fundamental part of how we perceive the world. However, current artificial intelligence models that process audio, known as audio large language models (ALLMs), often struggle with this complex task. They tend to rely on basic sound cues and make decisions in a single step, which limits their accuracy in figuring out direction and distance, and makes it hard to understand how they arrive at their conclusions.
While some models, like BAT, have shown progress in answering spatial questions using binaural audio (sound recorded as if heard by two ears), they often use very broad categories like ‘left’ or ‘right’ and don’t explicitly consider the geometry of the environment. This means they lack the precision and robustness needed for more advanced tasks.
Introducing OWL: A New Approach to Spatial Audio Reasoning
Researchers have introduced a new framework called OWL, which stands for Geometry-Aware Spatial Reasoning for Audio Large Language Models. OWL aims to overcome these limitations by integrating a novel geometry-aware audio encoder called SAGE with a spatially grounded ‘chain-of-thought’ (CoT) reasoning process. This allows OWL to not only detect sounds but also to localize them with much greater accuracy and provide interpretable reasoning for its spatial understanding.
SAGE: The Geometry-Aware Audio Encoder
At the heart of OWL is SAGE, a unique audio encoder that learns to connect binaural acoustic features with the 3D structure of a space. During its training, SAGE uses panoramic depth images (which capture the 3D layout of a room) and simulated room-impulse responses (how sound bounces around a specific environment). This ‘privileged supervision’ helps SAGE understand how the environment’s geometry influences sound propagation. Crucially, once trained, SAGE only requires audio input during inference, making it practical for real-world applications.
OWL’s Spatially Grounded Chain-of-Thought
Building on SAGE’s robust representations, OWL employs a spatially grounded chain-of-thought mechanism. Instead of simply providing an answer, OWL breaks down complex acoustic queries into smaller, interpretable steps. For example, it can rationalize that ‘sound A at 8 o’clock is left of sound B at 1 o’clock.’ This multi-step reasoning, developed through a curriculum learning approach, enables OWL to achieve ‘o’clock-level’ azimuth and direction-of-arrival (DoA) estimation, which is far more precise than previous methods.
BiDepth: A New Dataset for Training and Evaluation
To facilitate the large-scale training and evaluation of OWL, the researchers created and released BiDepth. This extensive dataset contains over one million question-answer pairs, combining binaural audio with panoramic depth images and room impulse responses. BiDepth covers both in-room and out-of-room scenarios and includes four types of questions:
- Event Detection: Identifying sound sources.
- Direction Estimation: Pinpointing azimuth, elevation, and distance.
- Spatial Reasoning: Answering relational queries like ‘Is source 1 left of source 2?’
- CoT for Spatial Reasoning: Providing step-by-step rationales for spatial conclusions.
This comprehensive dataset provides the explicit geometric supervision necessary for training models like SAGE and OWL.
Impressive Results and Performance
OWL’s performance has been rigorously tested on two benchmark datasets: the new BiDepth and the public SpatialSoundQA. The results are compelling:
- SAGE alone reduces the mean Direction-of-Arrival (DoA) error by 11 degrees and significantly decreases the distance error rate compared to state-of-the-art methods.
- OWL improves spatial reasoning question-answering accuracy by up to 25% over BAT, a leading baseline.
- It consistently outperforms both open-source models (like VideoLLaMA2, RAVEN, and AudioFlamingo2) and even closed-source high-capacity LLMs like Gemini-1.5-Pro, Gemini-2.5-Pro, and Gemini-2.5-Flash across various tasks, especially in reasoning-heavy scenarios.
- The chain-of-thought supervision further boosts reasoning accuracy by over 11% and provides consistent gains in detection and DoA.
Ablation studies confirmed that the geometric loss in SAGE is crucial for improving localization, and the multi-stage curriculum learning is essential for building robust spatial reasoning capabilities in OWL.
Also Read:
- Structured Emotion Graphs Enhance AI’s Understanding of Speech Emotion
- Navigating 3D Environments with Integrated Audio-Visual Intelligence
Future Directions
While OWL represents a significant step forward, the researchers acknowledge that BiDepth is currently simulation-based. Future work will focus on extending BiDepth to include real-world recordings to test robustness in more complex acoustic conditions. Additionally, the current reasoning tasks are single-turn, and expanding to multi-turn, interactive dialogues, and integrating richer grounding with vision or inertial sensing are promising avenues for future research. These advancements position OWL as a foundational step toward creating embodied AI agents capable of human-like spatial reasoning.
For more technical details, you can refer to the full research paper: OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models.
