TLDR: A new study introduces the Adversarial PhoneTic Prompting (APT) attack, showing that AI music (SUNO, YuE) and video (Veo 3) generation models can reproduce copyrighted content by preserving phonetic structure in lyrics, even when semantic meaning is altered. This “sub-lexical memorization” and “phonetic-to-visual regurgitation” raise significant concerns for copyright and content originality in generative AI.
A new research paper titled “Bob’s Confetti: Phonetic Memorization Attacks in Music and Video Generation” by Jaechul Roh, Zachary Novack, Yuefeng Peng, Niloofar Mireshghallah, Taylor Berg-Kirkpatrick, and Amir Houmansadr, reveals a surprising vulnerability in advanced AI models that generate music and video. These models, designed to create content from text, can “memorize” and reproduce copyrighted material even when given seemingly altered inputs. This phenomenon, termed Adversarial PhoneTic Prompting (APT) attack, highlights a critical concern for copyright, safety, and content originality in the age of generative AI.
The Adversarial PhoneTic Prompting (APT) Attack
The core of this research is the APT attack, a novel method where the lyrics provided to a generative AI model are semantically changed, but their acoustic structure, like rhyme and rhythm, is preserved through homophonic substitutions. Imagine changing “mom’s spaghetti” to “Bob’s confetti” – the meaning is completely different, but the sound pattern remains strikingly similar. The researchers found that despite these significant semantic distortions, models like SUNO and YuE regenerate outputs that are remarkably similar to their original training content.
This vulnerability, which they call “sub-lexical memorization,” was observed across various audio metrics, including CLAP, AudioJudge, and CoverID. It persisted across multiple languages and musical genres, demonstrating a widespread issue. For instance, phonetically modified versions of “Jingle Bell Rock” achieved high similarity scores in melody and rhythm, even with altered lyrics like “Giggle shell, Giggle shell, Giggle shell sock.” Similarly, rap songs like Eminem’s “Lose Yourself” and Kendrick Lamar’s “DNA” also showed strong musical resemblance despite significant lyrical changes such as “cheese weak” for “knees weak” or “BMA” for “DNA.”
Phonetic-to-Visual Regurgitation in Video Generation
Even more surprisingly, the study extended its findings to text-to-video (T2V) models. When prompted with phonetically modified lyrics from Eminem’s “Lose Yourself,” the Veo 3 model reconstructed visual elements from the original music video. This included details like the character’s appearance (a hooded male figure) and scene composition (dimly lit urban environments), despite the prompt containing no visual cues. The researchers named this “phonetic-to-visual regurgitation,” indicating that phonetic patterns alone can trigger the recall of memorized audiovisual content.
This suggests that these multimodal generative systems, which synthesize human speech along with other modalities (like background music or video frames), are susceptible to more than just exact lyric reproduction. They can be influenced by subtle phonetic cues that unlock memorized content, raising urgent questions about how these systems handle copyright, ensure safety, and maintain content provenance.
Also Read:
- Unmasking Hidden Memorization in Text-to-Image AI
- Unmasking LLM Inputs: The Inversion Threat to Internal States
Implications and Future Considerations
The research suggests that this memorization occurs because lyrics and rhythm play a central role in the structure of many songs, especially in genres like rap and Christmas music. When the phonetic structure is mimicked, even with nonsensical phrases, the models activate memorized patterns tied to rhythm, syllabic stress, or acoustic cadence. This implies that models might prioritize sub-lexical timing and sound features more heavily than semantic meaning during training.
However, the attack was not universally successful. Melody-first compositions or songs with less rigid lyrical timing, such as many K-pop tracks, were more resistant. This indicates that memorization is most easily triggered when lyrics are the dominant carrier of rhythmic and structural information.
The findings underscore the need for new evaluation and safety frameworks for generative AI. As models like SUNO, YuE, and Veo 3 become more prevalent, ensuring originality and copyright compliance will require defenses that account for the subtle power of phonetic cues, not just direct textual or visual similarity. For more details, you can read the full research paper here.
