TLDR: This research introduces a new approach to audio fingerprinting by using pretrained music foundation models (like MuQ and MERT) as neural network backbones and expanding data augmentation during training. The study demonstrates that these models significantly outperform existing state-of-the-art methods and classical baselines in identifying music, even when heavily distorted or compressed, across both track and segment levels.
In today’s digital age, where music is constantly shared, remixed, and manipulated across platforms like TikTok, identifying the original source of a musical recording has become increasingly challenging. This challenge has spurred the development of more robust audio fingerprinting techniques, which are essential for applications ranging from music identification to broadcast monitoring.
A recent research paper, titled “Robust Neural Audio Fingerprinting using Music Foundation Models,” delves into innovative methods to enhance the resilience of audio fingerprinting. The authors, Shubhr Singh, Kiran Bhat, Xavier Riley, Benjamin Resnick, John Thickstun, and Walter De Brouwer, introduce two significant advancements in this field.
Firstly, their work leverages the power of pretrained music foundation models, such as MuQ and MERT, as the core architecture for neural audio fingerprinting. These models, already trained on vast amounts of musical data, bring a sophisticated understanding of music structure to the fingerprinting process. Secondly, the researchers significantly expand the use of data augmentation during training. This involves exposing the fingerprinting models to a wide array of audio manipulations, including time stretching, pitch modulation, various compression techniques, and filtering. This rigorous training regimen helps the models learn to identify music even when it’s heavily altered.
The team systematically evaluated their new techniques against two leading neural fingerprinting models, NAFP and GraFPrint, as well as a Shazam-like baseline implemented with the Dejavu library. The results were compelling: fingerprints generated using music foundation models like MuQ and MERT consistently outperformed models trained from scratch or those pretrained on general, non-musical audio. This highlights the immense value of specialized pretraining in the musical domain.
Specifically, the unfrozen MuQ model achieved the highest overall accuracy in track-level identification, demonstrating remarkable robustness, particularly against Encodec compression, a common challenge for other models. While these advanced models showed superior performance, the study also noted some areas for improvement, particularly their sensitivity to certain filtering-based audio augmentations.
Beyond identifying entire tracks, the research also explored segment-level evaluation, which is crucial for managing large music catalogs and pinpointing specific sections of a recording. Here too, the unfrozen MuQ model excelled, accurately localizing fingerprint matches within audio segments. This capability is vital for practical applications where only a portion of a track might be available or modified.
The methodology involved training on 300,000 samples from the Disco-10M dataset and evaluating on the FMA dataset for track-level tasks, and the Pexeso Audio Fingerprinting Benchmark Toolkit for segment-level tasks. The comprehensive set of augmentations ensured a thorough test of robustness against real-world audio degradations.
Also Read:
- Enhancing Robot Navigation in Extreme Environments with Multimodal AI
- Understanding Flow Matching: New Bounds on Distributional Error
In conclusion, this research strongly suggests that pretrained music foundation models offer a powerful new direction for audio fingerprinting. Their ability to maintain accuracy despite significant audio manipulations represents a substantial leap forward in identifying music across modern media platforms. Future work will aim to address the identified weaknesses and explore even more complex adversarial audio changes. You can read the full paper for more technical details here.
