TLDR: This research introduces two innovative AI pipelines that automatically generate music videos from any song, instrumental or vocal, using deep learning models. Inspired by human production workflows, the systems analyze audio for emotional cues and instrumental patterns, translating them into textual scene descriptions via language models, which then guide generative models to produce video clips. A preliminary user evaluation indicates promising potential for storytelling, visual coherence, and emotional alignment, expanding music visualization beyond traditional methods.
The world of music visualization is undergoing a significant transformation, moving beyond traditional, hand-crafted methods to embrace the power of artificial intelligence. A recent research paper, “From Sound to Sight: Towards AI-authored Music Videos,” explores novel approaches to automatically generate music videos from any song, whether it’s instrumental or features vocals.
Authored by Leo Vitasovic, Stella Graßhof, Agnes Mercedes Kloft, Ville V. Lehtola, Martin Cunneen, Justyna Starostka, Glenn McGarry, Kun Li, and Sami S. Brandt, this paper introduces two distinct AI-powered pipelines. These systems are designed to analyze audio, detect musical qualities like emotional cues and instrumental patterns, and then translate these into textual scene descriptions using advanced language models. Subsequently, generative models are employed to produce the corresponding video clips, which are then merged into a complete music video.
The core idea behind this research is to tap into the natural human tendency to associate music with other senses, particularly sight. Historically, from early visual accompaniments to live performances to the modern music video, the synergy between sound and vision has continuously evolved. With the rise of AI-generated art, which has moved from simple automated creations to complex works across various artistic domains, the potential for AI to author music videos is becoming a reality.
While existing AI tools can generate visuals from text prompts, they often require manual input for scene descriptions or are limited to music with lyrics. This new research aims to overcome these limitations by focusing on music-first video generation, capable of handling instrumental and non-verbal audio without relying on lyrics. The pipelines are inspired by the workflows of human music video producers, aiming for computational synaesthesia – the ability to transform sensory input from one modality (sound) into another (sight).
The Two AI Pipelines for Music Video Creation
The paper proposes two main pipelines, each leveraging different AI models for audio analysis and script generation:
Pipeline 1: The CLAP-based Approach
This pipeline begins by segmenting the input song into smaller audio clips. Each segment, as well as the entire track, is then analyzed using Contrastive Language-Audio Pre-training (CLAP). CLAP is a powerful model that aligns audio signals with natural language descriptions, allowing it to extract high-level musical attributes like ‘melodic piano,’ ‘upbeat tempo,’ or ‘sad and moody strings.’ These descriptors are then fed into a large language model (LLM), specifically DeepSeek-R1-Distill-Llama-8B, which acts as a “video script-writing tool.” The LLM generates concise, narrative-like scene prompts, ensuring a coherent storyline that aligns with the music’s mood and style. Finally, these textual prompts are converted into video clips using a diffusion-based text-to-video model called mochi-1, and the clips are assembled with the original audio.
Pipeline 2: The LALM-based Approach
This alternative pipeline takes a more integrated approach. Instead of explicit feature extraction, it directly feeds the raw audio track into a Large Audio Language Model (LALM). The LALM is prompted to generate a coherent narrative concept or short story that thematically and emotionally resonates with the song. This story then serves as the blueprint for the entire music video. The LALM itself, or a reasoning LLM, breaks down this overarching narrative into segment-aligned visual cues. These scene descriptions are then used to generate video clips via another diffusion-based text-to-video model, WAN 2.1, before final assembly.
Both pipelines share common components, such as audio segmentation (either randomized or rule-based to mimic human editing) and the final video generation and assembly steps. The goal is to produce videos that are not just random visual montages but are guided by a coherent storyline or thematic arc, reflecting the emotional landscape of the audio.
Also Read:
- Think2Sing: Advancing Expressive 3D Head Animation for Singers
- AImoclips: Measuring How AI Music Conveys Feelings
Evaluation and Future Directions
To assess the generated music videos, the researchers conducted experiments using a diverse set of music genres, including Irish folk, blues, jazz, heavy metal, and classical music. A preliminary user study with five participants evaluated aspects like storytelling, visual impression, transitions, emotional consistency, and overall impression. While the CLAP-based pipeline generally received slightly higher ratings, the results are considered exploratory due to the small sample size.
Qualitative feedback from participants, including a professional videographer, highlighted areas for improvement. A recurring critique was the lack of visual and narrative consistency, with inconsistent color palettes, lighting, and character appearances across scenes. However, participants noted that the AI was often successful at capturing the general mood and theme of the music without lyrics, indicating the promise of the core concept.
The limitations identified, particularly the struggle of text-to-video models to maintain visual consistency for recurring characters, point to clear directions for future work. These include improving character consistency, incorporating lyrics for deeper narrative understanding, exploring human-AI creative collaboration, and conducting larger, more in-depth user studies. The code and generated music videos are publicly available on their Github repository, inviting further exploration and development.
This research underscores the significant potential of integrating latent audio feature extraction with LLM-driven text-to-video generation to create compelling and conceptually coherent music videos, pushing the boundaries of AI in artistic expression.
