TLDR: This research introduces the Expressive Music Variational AutoEncoder (XMVAE), a novel AI model designed to generate classical piano performances from scratch. It uses a new Expressive Compound Word (ECP) representation to capture both musical scores and expressive performance details. XMVAE features two branches, a ‘Composer’ (VQ-VAE) for score content and a ‘Pianist’ (vanilla VAE) for expressive nuances, trained jointly. Evaluations show XMVAE generates classical performances with superior musical quality compared to existing models, bridging the gap between AI and human-like musical expression.
Creating classical piano performances that capture both the composer’s original intent and a performer’s expressive nuances has long been a complex challenge in artificial intelligence. Traditional methods often struggle to balance the structured elements of a musical score with the subtle, human-like variations in timing, dynamics, and articulation that make a performance truly engaging.
This new research introduces a novel approach to tackle this dual challenge, aiming to build AI systems that can act as both composer and pianist. The core of their innovation lies in two key components: a new way to represent music and a sophisticated AI model designed to process it.
First, the researchers developed the Expressive Compound Word (ECP) representation. Unlike older methods that might miss the intricate rhythmic structures or expressive details of classical music, ECP is designed to capture both. It encodes music into ‘compound tokens,’ where each token simultaneously describes score-related features (like notes and metrical structure) and performance-related details (such as how long a note is held, its velocity, and timing variations). This comprehensive representation allows the AI to understand the music in a much richer way.
Building on ECP, the team then proposed the Expressive Music Variational AutoEncoder (XMVAE). This model is ingeniously structured with two distinct branches, each mimicking a different aspect of musical creation. The ‘Composer’ branch, based on a Vector Quantized Variational AutoEncoder (VQ-VAE), focuses on generating the fundamental score content and metrical structure. It’s like the part of the AI that understands the blueprint of the music. The ‘Pianist’ branch, a vanilla VAE, then takes this score information and adds the expressive details, learning a global representation of performance style. This branch is responsible for the ‘how to play’ aspect, injecting the human-like feel into the performance.
These two branches are trained together, ensuring that the composition and expressive elements are seamlessly integrated. The XMVAE also incorporates advanced features like a multiscale encoder, which helps it understand musical context at different levels, and an orthogonal Transformer decoder for efficient processing of the complex ECP tokens.
To evaluate their creation, the researchers conducted extensive tests, both objective and subjective. They compared XMVAE against several existing state-of-the-art models using a large dataset of classical piano performances. The results were highly promising: XMVAE consistently generated classical performances that were superior in musical quality. Notably, pre-training the Composer branch on additional musical score datasets further enhanced the model’s performance, demonstrating the benefit of exposing the AI to a wider range of musical compositions.
Also Read:
- Unpacking Music Recommendations: The Role of Content Filtering
- Self-Supervised Learning Guides Robots to Bio-Inspired Movement
In subjective listening tests, human participants rated XMVAE’s generated music higher in terms of coherency, richness, and overall quality compared to other AI models. While there’s still a gap when compared to human-composed and performed music, this research marks a significant step forward in the field of AI-generated expressive classical piano performances. The potential applications are vast, from creating virtual performers to providing enhanced creative tools for musicians. You can read the full research paper here.
