TLDR: A new model, BeatFCOS, redefines beat and downbeat tracking in music as a temporal object detection task. It adapts the FCOS computer vision detector to 1D audio, using WaveBeat’s feature extractor and a Feature Pyramid Network. Instead of traditional Dynamic Bayesian Networks, BeatFCOS employs Non-Maximum Suppression (NMS) with a data-driven threshold, simplifying post-processing. The model also introduces a “leftness” score to focus on the start of beat intervals. It achieves competitive results on standard music datasets, demonstrating the effectiveness of object detection techniques for musical rhythm analysis.
A groundbreaking new approach to beat and downbeat tracking in music, traditionally a complex task in music information retrieval, has been introduced by Jaehoon Ahn and Moon-Ryul Jung from Sogang University. Their research reframes the problem of identifying musical beats as a temporal object detection task, drawing inspiration from techniques commonly used in computer vision for detecting objects in images.
Historically, beat tracking models have relied on machine learning techniques like recurrent neural networks (RNNs), long-short-term memory (LSTM) networks, temporal convolutional networks (TCNs), and Transformers. These models typically produce frame-level activations, indicating the likelihood of a beat at a given moment. However, a crucial post-processing step often involves Dynamic Bayesian Networks (DBNs) to refine these activations into a final sequence of beat positions. DBNs, while effective, can be computationally intensive and sometimes struggle with changes in tempo or time signature.
The new model, named BeatFCOS, is an adaptation of the FCOS (Fully Convolutional One-Stage Object Detection) detector, a prominent architecture in computer vision. Instead of processing 2D image data, BeatFCOS is designed to work with 1D audio waveform data. A key innovation is the representation of beats and downbeats not as single points in time, but as temporal “intervals.” For instance, a beat interval spans two consecutive beats, providing crucial information about the distance between them.
To achieve this, the researchers replaced FCOS’s original image-based backbone with WaveBeat’s temporal feature extractor, a model known for its spectrogram-free approach and direct processing of raw audio. They also integrated a Feature Pyramid Network (FPN) to effectively capture multi-scale temporal patterns within the audio. This allows the model to detect beats and downbeats across different rhythmic complexities.
One of the most significant departures from traditional methods is BeatFCOS’s post-processing. Instead of DBNs, it employs Non-Maximum Suppression (NMS), a standard technique in object detection. NMS works by selecting the highest-scoring predicted interval and then suppressing any highly overlapping, lower-scoring predictions. This process is simpler and less heuristic than DBNs. The researchers also developed a data-driven method to determine the optimal Intersection-over-Union (IoU) threshold for NMS, making the process more principled and less reliant on manual tuning. Furthermore, they adopted Soft-NMS, a variant that gradually decays scores of overlapping boxes rather than outright removing them, which helps retain valid, closely-spaced beat predictions.
Another novel aspect is the introduction of a “leftness” score, which replaces the “centerness” score used in the original FCOS. This modification emphasizes the left edge of the beat interval, aligning with the understanding that the beat itself occurs at the start of the interval. This explicit guidance helps the model localize beats more accurately.
The BeatFCOS model was trained using the Adam optimizer on a diverse set of standard music datasets, including Ballroom, Hainsworth, Beatles, RWC Popular, GTZAN, and SMC. Evaluations showed that BeatFCOS achieves competitive results, particularly in downbeat tracking, and often outperforms the peak-picking variant of WaveBeat. While it doesn’t consistently surpass all previous systems in every metric, it demonstrates a compelling new framework for beat tracking that simplifies the modeling pipeline.
Also Read:
- MotionBeat: Advancing Music Understanding Through Motion-Aligned Learning
- MimicParts: Advancing Realistic 3D Motion Generation from Speech
This research highlights the potential of adapting computer vision techniques to audio processing, offering a more transparent and less ad hoc solution for rhythmic event prediction. The full research paper can be found here: Beat Tracking as Object Detection.
