TLDR: This research explores aligning code and audio embedding spaces to improve LLM-powered music code generation. Finding no simple linear relationship, the authors developed a neural network model that learns to predict audio embeddings from code embeddings. By augmenting a small dataset and using contrastive learning, their model significantly improved the alignment between code and audio representations, enabling LLMs to better understand and generate musically diverse output.
Creative coding, especially live-coding, is a dynamic field where artists use code to generate multimedia, often in real-time performances. This unique art form, highlighted by the TOPLAP live coding manifesto, emphasizes the creation process itself. Tools like Sonic Pi, SuperCollider, and Tidal Cycles are popular domain-specific languages (DSLs) that allow performers to translate musical ideas into code efficiently. This research focuses on Sonic Pi, a Ruby-based live coding DSL widely used for education and artistic expression.
Large Language Models (LLMs) offer exciting possibilities for creative coding by reducing the burden of syntax, allowing artists to concentrate on creative concepts. However, current code generation systems face a significant challenge: they often fail to provide diverse and functionally unique code candidates, particularly in multi-modal scenarios where the output is sound or video, not just text. LLMs typically evaluate code based on text similarity, which doesn’t account for perceptual differences in audio. This means multiple code snippets might look different but sound identical, hindering their usefulness in live coding where output must align with the coder’s musical intention.
To address this, the researchers explored the relationship between code and audio by investigating the topology of their embedding spaces. Embedding models, common in NLP, map data like code or audio into high-dimensional vectors where similar items are close together. The idea is that if an alignment map could be built between code and audio embeddings, it would allow reasoning about musical similarity based solely on the generated code.
Their preliminary investigation involved creating a dataset of 28 Sonic Pi code-audio pairs. They extracted code embeddings using `distilroberta-base` and audio embeddings using Meta’s `wav2vec2` model. Analyzing the cosine similarity between these pairs, they found no simple linear relationship between code and audio embeddings. This suggests that a direct, straightforward mapping is not possible, which aligns with the complexity of a Sonic Pi compiler and audio engine.
To train a model for this complex relationship, a larger dataset was needed. Since manual creation was impractical, they augmented their initial 28 examples using Jinja templates. By randomizing parameters like sample type, synthesizer, notes, and timers, they generated 200 different code files for each example, resulting in 5600 initial files. After filtering for valid audio generation, they ended up with 5400 code-audio pairs. This automated process, using the Python-Sonic interface, was crucial for building a sufficiently large dataset.
The core of their solution is a neural network model designed to predict the distance between two audio embeddings based on their respective code embeddings. This model doesn’t synthesize audio directly but aims to understand how changes in code embeddings affect the perceptual distance in audio. They employed a symmetric architecture with two independent Multi-Layer Perceptrons (MLPs), one for code and one for audio embeddings. These MLPs project the pre-trained embeddings into a common latent space.
For training, they used the InfoNCE loss, a contrastive learning objective. This method helps align semantically related code-audio pairs by pulling them closer in the shared embedding space while pushing apart mismatched pairs. This self-supervised approach is ideal because explicit labels for “musical similarity” are not readily available.
The model’s performance was evaluated using Canonical Correlation Analysis (CCA) and Centered Kernel Alignment (CKA). CCA measures linear correlation, while CKA captures more general structural similarities. Before training, the CKA was very low (0.090) and CCA was also low (0.140), indicating minimal initial correlation. After training, the best configuration achieved a CKA of 0.590 and a normalized CCA of 0.902. This represents a significant improvement, demonstrating that the model successfully learned a meaningful shared embedding space. This means that given a code embedding, the model can reliably approximate the corresponding audio embedding, even without perfect prediction.
Also Read:
- Interpretable AI Models Show Enhanced Robustness in Music Emotion Recognition
- EmoSteer-TTS: Precise Emotion Control in Synthesized Speech Without Retraining
In conclusion, this research demonstrates that a neural network model can successfully learn to align code and audio embeddings, providing a mechanism for LLMs to better understand and generate musically diverse code. This work, detailed further in their research paper available at this link, bridges the gap between what a code generation model “writes” and what a user “hears,” paving the way for more intuitive and musically intelligent creative coding tools.
