TLDR: A large-scale empirical study identifies that combining self-supervised pre-training with supervised post-training on diverse bioacoustic and general audio data yields state-of-the-art, generalizable bioacoustic encoders. The research also introduces new evaluation benchmarks for individual identification and vocal repertoire discovery, emphasizing the importance of data diversity for robust model performance.
The study of sounds produced by living organisms, known as bioacoustics, is crucial for understanding animal behavior, monitoring biodiversity, and aiding conservation efforts. Many tasks in this field, such as identifying species, individuals, or behaviors, are well-suited for machine learning. However, a common challenge is the limited availability of annotated data, which highlights the need for a versatile bioacoustic encoder capable of extracting useful representations for various downstream tasks.
Previous bioacoustic encoders have often been limited in scope, focusing on a narrow range of species, typically birds, or relying on a single model architecture or training method. Furthermore, their evaluation has usually been restricted to a small set of tasks and datasets. A recent large-scale empirical study, detailed in the paper “What Matters for Bioacoustic Encoding”, addresses these limitations by exploring aspects of bioacoustics that have been scarcely considered before.
This comprehensive study investigates the impact of training data diversity and scale, different model architectures and training approaches, and the breadth of evaluation tasks and datasets. The researchers achieved state-of-the-art encoders on both existing and newly proposed benchmarks. A key finding is that self-supervised pre-training, followed by supervised post-training on a mixed corpus of bioacoustics and general audio, yields the strongest performance, both within and outside of the training data distribution. The importance of data diversity at both stages of training was also highlighted.
Understanding the Approach
The study systematically examined four key components: model architectures, data-mixes, training paradigms, and an expanded evaluation methodology. For model architectures, they compared CNN-based and transformer-based models, along with supervised and self-supervised learning. Regarding data, they trained and evaluated models using a broader and more taxonomically diverse bioacoustic dataset than previous work, also assessing the impact of incorporating general audio data like AudioSet.
The researchers explored sequential training paradigms, or “training recipes,” involving pre-training and post-training with both self-supervised and supervised learning. They also assessed how non-bioacoustic audio data influenced different training stages. The models were evaluated across established benchmarks like BEANS and BirdSet, as well as newly curated datasets designed to test generalization to challenging real-world scenarios, providing a clearer picture of what enhances bioacoustic representation learning.
Key Discoveries
The study found that self-supervised models, under comparable training conditions, showed strong out-of-distribution generalization, even though they might underperform supervised models on in-distribution tasks. A significant improvement in model transferability was observed when general audio was included in bioacoustic training. By combining these insights, the researchers proposed training recipes and models that achieved overall state-of-the-art results on their extensive evaluation benchmark, offering a versatile encoder for bioacoustic research.
In contrast to some prior works, this study found that combining self-supervised and supervised learning on diverse data—including general audio during both pre-training and post-training—brought significant benefits when transferring these representations to detection and classification benchmarks. This suggests that these two learning paradigms are complementary for bioacoustic representation learning.
Also Read:
- Advancing Audio Fingerprinting with Pretrained Conformer Encoders
- Advancing Audio Understanding with Multi-Hypothesis Self-Supervised Learning
Expanding Evaluation
Beyond model development, the study also broadened bioacoustic evaluation. They curated new benchmarks for individual identification and vocal repertoire classification from public datasets. Additionally, they augmented existing evaluation suites with retrieval and clustering metrics. These additions allow for a direct assessment of representation quality and are aligned with practical tasks such as audio-to-audio retrieval and discovering an animal’s vocal repertoire. The findings indicate that large-scale bioacoustic pre-training is an effective way to achieve representations that generalize well to these less-studied tasks.
The researchers plan to release the model checkpoints to support ongoing research and application, hoping to accelerate advancements in animal communication and conservation through bioacoustics.
