TLDR: This research introduces a novel deep learning architecture for musical agents that learns musical relationships between separate audio tracks (stems) from examples, rather than relying on pre-defined rules. Comprising a Perception module (Wav2Vec 2.0) for converting audio to symbolic sequences, a Decision module (Transformers) for learning and generating relationships, and an Action module (concatenative synthesis) for audio output, the system demonstrates the ability to learn general musical correspondences from multi-stem datasets. This approach aims to enhance the customizability and versatility of musical agents for creative applications.
Imagine a musical partner that doesn’t just follow pre-programmed rules but actually learns how different musical parts interact, allowing it to create new music in real-time based on your input. This is the exciting frontier explored in the research paper titled “Learning Relationships between Separate Audio Tracks for Creative Applications” by Balthazar Bujard, Jérôme Nika, Fédéric Bevilacqua, and Nicolas Obin.
The core idea behind this research is to move beyond traditional musical agents that rely on manually defined rules for generating music. Instead, the authors propose a system that can automatically learn complex musical relationships from existing music examples. This learning process aims to enable musical agents to respond more creatively and flexibly to live musical inputs.
The proposed system is built around a three-module architecture:
The Perception Module
This module acts as the system’s ‘ears.’ It takes a continuous audio signal, like a live performance, and transforms it into a sequence of discrete symbols, much like words in a language. To do this, it uses a sophisticated audio encoder called Wav2Vec 2.0, which is pre-trained on music. This allows the module to capture high-level musical information, not just basic sound characteristics, and represent it in a way the rest of the system can understand.
The Decision Module
This is the ‘brain’ of the musical agent. Once the Perception Module has converted the input audio into a symbolic sequence, the Decision Module steps in. It’s designed to learn and exploit the relationships between these symbolic sequences. For instance, if it’s trained on many songs where a bassline (Track A) typically interacts with a drum beat (Track B) in a certain way, it learns that relationship. When given a new bassline, it can then generate a symbolic ‘specification’ for a coherent drum beat. This module uses a Transformer-decoder architecture, a type of neural network known for its ability to process sequences and learn complex patterns.
Also Read:
- Unveiling Music AI Decisions: Introducing MUSE-Explainer for Interpretable Music Analysis
- Unlocking New Abilities: How Reinforcement Learning Helps Language Models Compose Skills
The Action Module
Finally, the Action Module is the ‘voice’ of the agent. It takes the symbolic specification generated by the Decision Module and converts it back into actual audio. This is achieved through a technique called corpus-based concatenative synthesis. Essentially, it searches a pre-defined library of audio segments (a ‘corpus’) for pieces that match the symbolic specification and then stitches them together to create the musical output. The system also includes a ‘constrained generation’ method to ensure that the generated symbols correspond to available audio segments in the corpus, leading to more musically relevant outputs.
To train and evaluate this system, the researchers used multi-stem datasets like MoisesDB (contemporary music) and MICA (free improvisations). These datasets are crucial because they contain separate audio tracks (stems) from the same musical piece, allowing the system to learn how different instruments or vocal parts relate to each other. The evaluation involved a ‘re-generation task,’ where the system was asked to predict the symbolic sequence of one track (B) given another related track (A) from the same original mix. While perfect re-generation wasn’t the goal, the results showed that the models significantly outperformed random baselines, confirming that they successfully learned general musical relationships within the datasets. Smaller symbolic vocabularies and the constrained generation technique generally led to better performance.
The research acknowledges several limitations, including the inherent complexity of musical relationships, the relatively small size of available multi-track datasets, and the challenge of interpreting what specific musical information the Perception Module encodes. Future work aims to explore ‘creative generalization’ – the ability to apply learned relationships to entirely new musical contexts – and to incorporate qualitative evaluations with musicians to assess the system’s creative potential.
This work represents a significant step towards creating more adaptable and musically intelligent agents that can learn from examples rather than being explicitly programmed, opening new avenues for creative applications in music. You can read the full paper here: Learning Relationships between Separate Audio Tracks for Creative Applications.
