TLDR: JAM is a new, compact AI model for generating songs from lyrics. It uses a flow-matching approach and offers unprecedented fine-grained control over word-level timing and song duration, which is crucial for musicians. JAM also incorporates aesthetic alignment to produce higher-quality, human-preferred songs. Despite being smaller than other models, it achieves superior lyrical fidelity and overall musical quality, addressing key limitations in existing lyrics-to-song generation systems.
Artificial intelligence has made significant strides in various creative fields, and music generation is no exception. While AI can now produce high-quality audio, creating full songs with vocals that offer fine-grained control and aesthetic appeal remains a complex challenge. Existing models often struggle with precise timing, lyrical accuracy, and overall musicality, leading to outputs that may not fully satisfy professional musicians or discerning listeners.
Introducing JAM: A New Era of Controllable Song Generation
A new research paper introduces JAM, a novel AI model designed to overcome these limitations. JAM stands out as a compact, flow-based song generator that provides fine-grained control over song creation, including word-level timing and overall duration. This level of control is a significant leap forward, allowing musicians to shape the prosody and rhythm of generated vocals with unprecedented precision.
JAM is built on a flow-matching architecture, making it efficient and capable of generating high-fidelity audio. Despite being considerably smaller than other leading song generation models, JAM achieves state-of-the-art performance across various objective and subjective evaluation metrics. For instance, it boasts significantly lower word error rates (WER) and phoneme error rates (PER), meaning the generated lyrics are clearer and more accurately aligned with the music.
Key Innovations and Features
The researchers highlight several key contributions that make JAM a powerful tool:
- Compact Architecture: At just 530 million parameters, JAM is less than half the size of its closest competitors, enabling faster inference and requiring fewer computational resources.
- Fine-Grained Alignment: JAM accepts word- and phoneme-level timing inputs, giving users precise control over the placement of each vocal sound. This enhances rhythmic flexibility and expressive timing.
- Enhanced Lyric Fidelity: The precise alignment capabilities drastically reduce word and phoneme error rates, ensuring that the generated vocals accurately reflect the input lyrics.
- Global Duration Control: Users can specify the overall song length and the spacing between vocal phrases, providing full control over the song’s structure, including instrumental introductions and outros.
- Rigorous Evaluation: To ensure fair comparison and avoid data contamination, the team introduced JAME, a public, genre-diverse evaluation dataset comprising newly released songs.
- Aesthetic Alignment: JAM incorporates Direct Preference Optimization (DPO) with automated song-quality models like SongEval. This iterative process refines the model to align its outputs with human aesthetic preferences, leading to more natural and enjoyable songs without the need for manual data annotation.
How JAM Works
JAM’s training involves three stages: pre-training on shorter song clips, fine-tuning for full-song generation, and a crucial preference alignment stage using DPO. The model takes lyrics with word-level timing, a style prompt (either text or audio), and a target duration as conditions. It learns to generate full songs at a high audio quality (44.1kHz) for durations up to 3 minutes and 50 seconds.
A significant challenge in practical application is providing the precise word-level timing information that JAM requires. While this is available during training, it’s difficult for non-experts to provide during inference. The researchers explored using AI models like GPT-4o for naive duration prediction and also investigated direct quantization of word-level timestamps to simplify user input. These experiments revealed that accurate timing predictions are critical for quality, and that training the model on quantized data could help mitigate artificiality when using simplified timing inputs.
Also Read:
- Music Arena: A New Approach to Evaluating Text-to-Music Models
- HH-Codec: A Breakthrough in Ultra-Low Bandwidth Speech Compression
Future Directions and Impact
While JAM represents a significant advancement, the researchers acknowledge the need for further development, particularly in creating an integrated duration predictor that can automatically estimate word- or phoneme-level timings from lyrics and melody. This would make JAM more accessible to non-expert users and enhance its robustness in real-world scenarios.
JAM is released as the first open-source model under PROJECT JAMIFY, aiming to foster academic research and creative exploration in AI-generated songs. It is intended for non-commercial, academic, and entertainment purposes, with a strong emphasis on ethical use and respecting intellectual property rights. This work paves the way for more efficient, controllable, and high-fidelity AI-assisted composition for both professionals and enthusiasts. You can find the full research paper here: JAM: A Tiny Flow-based Song Generator with Fine-grained Controllability and Aesthetic Alignment.
