TLDR: MIDI-VALLE is a new neural codec language model that significantly improves expressive piano performance synthesis. Adapted from the VALLE text-to-speech framework, it encodes both MIDI and audio as discrete tokens and is trained on a diverse dataset, allowing it to generate more realistic and expressive piano audio. It outperforms traditional methods by better generalizing across different musical styles and recording environments, as demonstrated by objective metrics and listening tests, though it currently struggles with complex genres like jazz.
Creating realistic and expressive piano performances from musical scores has long been a complex challenge in the world of artificial intelligence and music. Traditional methods often involve a two-step process: first, converting a music score into a digital representation (MIDI) that includes expressive details, and then synthesizing that MIDI into actual audio. However, these conventional synthesis models frequently struggle to adapt to different MIDI sources, musical styles, or recording environments, leading to less natural and expressive outputs.
Introducing MIDI-VALLE: A New Approach to Piano Performance Synthesis
To overcome these limitations, researchers have introduced MIDI-VALLE, a groundbreaking neural codec language model. This innovative model is an adaptation of the VALLE framework, which was originally designed for zero-shot personalized text-to-speech synthesis. MIDI-VALLE specifically targets piano MIDI-to-audio synthesis, aiming to produce highly expressive and realistic piano performances.
One of the key advancements of MIDI-VALLE is its ability to condition synthesis on a reference audio performance and its corresponding MIDI. Unlike previous systems that relied on piano rolls—a common but often limited way to represent musical notes—MIDI-VALLE encodes both MIDI and audio as discrete tokens. This token-based approach allows for a more consistent and robust modeling of piano performances, capturing subtle nuances that traditional methods might miss.
Enhanced Generalization Through Diverse Training
The model’s ability to generalize across various musical contexts is significantly boosted by its training on an extensive and diverse piano performance dataset called ATEPP. This dataset features recordings captured in a wide range of acoustic settings, allowing MIDI-VALLE to learn from a broader spectrum of musical expressions. This contrasts with older models often trained on more homogeneous datasets like Maestro, which, while valuable, lacked acoustic variety.
The tokenization process itself is a crucial element of MIDI-VALLE. For audio, it uses a fine-tuned version of the Encodec model, called Piano-Encodec, to convert audio performances into discrete tokens while preserving high-fidelity acoustics and timbral characteristics. For MIDI, it employs the Octuple MIDI tokenization method, which represents musical features like pitch, velocity, duration, and timing as distinct tokens. This method offers advantages over piano-roll representations by providing higher resolution and flexibility for capturing subtle timing variations essential for expressive articulation.
Also Read:
- FreeAudio: Crafting Precise and Extended Audio from Text Without Retraining
- Boosting Speech AI Performance Through Smart Data Generation
Impressive Performance and Future Potential
Evaluations show that MIDI-VALLE significantly outperforms state-of-the-art baselines. In objective tests, it achieved over 75% lower Fréchet Audio Distance (FAD) on the ATEPP and Maestro datasets, indicating a much higher perceptual quality and realism of the generated audio. Listening tests further confirmed its superiority, with MIDI-VALLE receiving substantially more votes than the baseline model, demonstrating improved synthesis quality and better generalization across diverse MIDI inputs.
While MIDI-VALLE excels with classical piano music, it currently faces challenges with genres like jazz, which have more complex harmonic structures and rhythms. However, its ability to adapt to various recording environments and reconstruct acoustics that closely match provided audio prompts is a major step forward. The model’s zero-shot design also means that an audio prompt can influence the loudness and timbre of the generated audio, highlighting its adaptability to diverse acoustic environments.
The development of MIDI-VALLE represents a significant leap in music performance synthesis, offering a more robust and adaptable framework for generating expressive piano audio. Future work will focus on improving its generalization across musical genres and exploring the impact of model size and alternative audio codec models. For more technical details, you can read the full research paper here.
