TLDR: SonicMotion is a new AI model that generates dynamic 3D spatial audio, specifically First-Order Ambisonics (FOA), allowing for precise control over moving sound sources. It comes in two versions: one conditioned by text and another by text plus spatial parameters for higher precision. The research also introduces a new dataset of spatial audio-caption pairs with moving sources. Evaluations show SonicMotion matches state-of-the-art models in audio quality and semantic alignment, with the parametric model excelling in spatial accuracy.
Immersive experiences in virtual reality, augmented reality, cinema, and music are increasingly relying on spatial audio to create a sense of presence and realism. This technology allows listeners to perceive sounds as coming from specific directions in a 3D environment. However, the manual creation of spatial audio, particularly in formats like first-order Ambisonics (FOA), can be a time-consuming and complex process, often requiring specialized software to precisely pan sound sources to their desired locations.
While generative AI models have made significant strides in producing speech, sound clips, and music, many are limited to mono or stereo audio, failing to account for the spatial positioning or movement of sound sources. Some recent models have attempted to address this gap, but often with limitations. For instance, some can generate stereo audio with spatial awareness but are restricted to basic directional cues like ‘left’ or ‘front,’ unable to capture ‘behind’ or ‘above.’ Others can generate FOA audio with precise azimuth and elevation control, but only for static sound sources, meaning the sound’s position cannot change over time.
Addressing these challenges, a new research paper introduces SonicMotion: Dynamic Spatial Audio Soundscapes with Latent Diffusion Models. This innovative end-to-end model is designed to generate FOA audio with precise control over moving sound sources in a 3D space. The researchers, Christian Templin, Yanda Zhu, and Hao Wang, have developed SonicMotion in two variations to cater to different user input needs and levels of precision.
The first variation, the ‘descriptive model,’ is conditioned primarily on a text prompt, allowing users to describe the desired soundscape and its spatial characteristics. The second, more precise variation, the ‘parametric model,’ takes both text input and specific spatial parameters, such as initial and final azimuth and elevation angles, and even movement speed. This dual approach offers flexibility, from natural language descriptions to highly detailed spatial control, helping to overcome the inherent ambiguities that can arise when describing complex spatial movements solely through text.
A significant contribution of this research is the creation of a new dataset specifically for spatial audio-caption pairs, which includes samples with both static and dynamically moving sound sources. Recognizing the lack of publicly available data for moving spatial audio, the team augmented three existing audio-caption datasets (Clotho, AudioCaps, and FreeSound). They converted mono audio signals into spatial audio by randomly generating initial and final azimuth and elevation angles, and for moving sources, determined whether azimuth, elevation, or both would change over time, along with the speed of movement. To ensure the captions accurately reflected these spatial dynamics, Google’s Gemma-3 12B model was leveraged to rephrase original captions with relevant spatial details.
At its core, SonicMotion utilizes a continuous diffusion transformer, a powerful generative architecture. Before this, an autoencoder compresses the 4-channel FOA audio into a more manageable latent space, and then decodes the generated latents back into the original audio format. This autoencoder was adapted from Descript Audio Codec (DAC) with modifications to better suit the FOA format and preserve spatial attributes during compression. The conditioning mechanism is crucial, with the parametric model incorporating a ‘state matrix’ that captures the sound source’s position over time, fused with the text embedding via a cross-attention module.
Evaluation of SonicMotion demonstrated its effectiveness. The autoencoder proved highly capable of accurately encoding and decoding spatial audio while preserving its spatial attributes. When compared against state-of-the-art text-to-audio models like AudioLDM 2 and Stable Audio Open, both SonicMotion variations maintained strong semantic alignment and audio quality, showing that the model doesn’t sacrifice general audio fidelity for spatial capabilities. Crucially, the parametric model significantly outperformed its descriptive counterpart in spatial accuracy, highlighting the benefit of explicit spatial parameters for precise control over dynamic sound sources.
Also Read:
- Latent Acoustic Mapping: A Self-Supervised Leap in Sound Localization
- Enhancing Music Mixing with Conversational AI Assistance
This work represents a notable step forward in generative spatial audio, offering a powerful tool for creating immersive soundscapes with dynamic elements. The researchers acknowledge that future work could explore extending SonicMotion to support multiple overlapping sound sources and control over the distance of the source, such as movement from far to near. For more technical details, you can refer to the full research paper available at arXiv.org.
