TLDR: This paper introduces a novel approach to self-supervised pitch estimation by integrating Optimal Transport (OT) into the training process. The proposed PESTO-OT model replaces complex equivariance loss functions with a single, more stable OT-based objective (1D Wasserstein distance). This method offers theoretical benefits like improved numerical stability and achieves competitive performance with state-of-the-art self-supervised pitch estimators, demonstrating the relevance of OT for translation-equivariant learning in audio analysis.
Pitch estimation, the process of identifying the fundamental frequency of sounds, is a crucial task in audio analysis, impacting fields from speech processing to music information retrieval. Traditionally, this has involved complex signal processing or training deep learning models on vast, meticulously labeled datasets. While highly accurate, these supervised deep learning methods, like CREPE, face significant hurdles: the immense effort and potential for error in labeling audio with millisecond precision, and their tendency to perform poorly when encountering sounds outside their training distribution (e.g., a model trained on music struggling with speech).
To overcome these limitations, self-supervised learning (SSL) has emerged as a powerful alternative. In SSL for pitch estimation, instead of relying on human-annotated labels, a system learns by observing how its own input changes. Specifically, a pair of audio frames are artificially pitch-shifted by a known amount. A neural network, often a Siamese network, is then trained to be ‘equivariant’ to this pitch shift. This means the network learns to produce outputs that also shift predictably in response to the input pitch shift. A Constant-Q Transform (CQT) is typically used as a pre-processing step, as it maps frequencies to a logarithmic scale where pitch shifts become simple translations, making the equivariance task more straightforward.
Previous self-supervised approaches include SPICE, which minimized an equivariance objective on scalar pitch predictions, and PESTO, which improved upon this by operating on estimated pitch distributions and utilizing a lightweight, transposition-equivariant architecture. The concept of translation equivariance extends beyond single pitch estimation, finding applications in tasks like multi-pitch, tempo, tonality, and key estimation.
Introducing Optimal Transport for Pitch Estimation
This new research simplifies the training of self-supervised pitch estimators like PESTO by introducing an objective based on Optimal Transport (OT). Optimal Transport provides a sophisticated framework for comparing probability distributions. Unlike simpler distance metrics, OT considers the ‘cost’ of transforming one distribution into another, effectively measuring the ‘horizontal displacement’ or ‘mass movement’ required. This concept is highly relevant to audio, as it can precisely measure frequency shifts between signals.
The proposed method, named PESTO-OT, replaces the multiple equivariance loss terms used in PESTO with a single, elegant 1D Wasserstein distance, a specific type of Optimal Transport metric. This new loss function compares the network’s output distribution for an original audio frame with the output distribution of its pitch-shifted version, after correcting for the known shift. This approach offers several theoretical advantages:
- It is symmetrical, meaning the comparison works the same regardless of which distribution is considered the ‘source’ or ‘target’.
- It is invariant to overall translations, ensuring that shifting both distributions by the same amount doesn’t change their measured distance.
- It scales linearly with the pitch shift, providing a clear and intuitive measure of the shift.
- Crucially, it enhances numerical stability by avoiding the large floating-point powers that can cause issues in previous loss functions.
Also Read:
- Mapping Sound: A U-Net Approach to Pinpointing Acoustic Sources
- ShrutiSense: Preserving the Nuances of Indian Classical Music
Performance and Future Outlook
The researchers evaluated PESTO-OT using the same rigorous protocol as PESTO, testing it across diverse datasets covering singing voice (MIR-1K), music (MDB-stem-synth), and speech (PTDB). Despite not undergoing extensive hyperparameter tuning, the new method achieved competitive performance compared to the state-of-the-art PESTO baseline, particularly when trained on the MIR-1K dataset. While a slight performance gap was observed when evaluating on the MDB dataset, which features a wider pitch range, the overall results are highly encouraging.
This work demonstrates the significant potential of using Optimal Transport-based loss functions for enforcing translation equivariance in self-supervised learning. The simplicity, theoretical soundness, and numerical stability offered by this approach make it a promising alternative for training advanced pitch estimators. The researchers believe that this contribution could have broad applications beyond single pitch estimation, extending to other Music Information Retrieval tasks that rely on translation equivariance, such as tempo, key, chord, and multi-pitch estimation. For more in-depth technical details, you can refer to the full research paper available here.
