TLDR: HarmonicRNN is a new deep linear RNN model designed for autoregressive generation of long music samples. It leverages efficient linear recurrences and novel pooling mechanisms to handle sequences up to one minute in length. The model achieves state-of-the-art log-likelihoods and perceptual metrics on audio datasets, demonstrating improved performance over previous models like SaShiMi and Mamba, particularly due to its use of CG-LRU, strided convolutions for pooling, and non-learned sinusoidal embeddings.
Generating long, high-quality audio waveforms in an autoregressive manner has long been a significant challenge in artificial intelligence. The sheer length of raw audio sequences and the intricate structures present across various timescales make this task particularly difficult. Traditional methods, including those based on recurrent neural networks (RNNs), causal convolutions, and even self-attention mechanisms, have shown limited success in truly mastering this domain.
However, recent advancements have brought deep state space models, also known as linear RNNs, into the spotlight for their efficiency in handling such complex sequences. Building on this progress, a new model called HarmonicRNN has been introduced, pushing the capabilities of linear RNNs for raw audio modeling. This innovative approach investigates different architectural choices and leverages context-parallelism, allowing it to train on audio sequences up to one minute in length, which translates to a remarkable 1 million tokens.
The core idea behind HarmonicRNN is to use linear recurrent layers, which are highly efficient. Unlike self-attention mechanisms that have a computational complexity that grows quadratically with sequence length (O(T^2)), linear recurrent layers maintain a linear complexity (O(T)). This makes them much more suitable for the extremely long sequences common in audio processing, especially during the generation (inference) phase where quadratic complexity becomes a major bottleneck.
HarmonicRNN draws inspiration from the SaShiMi model but introduces several key distinctions. It utilizes a specific type of recurrent layer called the Complex Gated Linear Recurrent Unit (CG-LRU) as its foundation. For managing different timescales within the audio, the model employs temporal down and up-pooling operations, using strided and dilated convolutions. A crucial finding was that setting the number of feature groups in these pooling convolutions to greater than one significantly improved stability. Furthermore, instead of learned embedding layers, HarmonicRNN uses non-learned sinusoidal embeddings, which were found to accelerate training convergence and improve final performance.
The model was rigorously tested on several audio datasets, including SC09 (spoken digits), Beethoven (piano sonatas), and YouTubeMix (piano playing). Notably, HarmonicRNN was trained on the full minute-long sequences of the YouTubeMix dataset, a feat that highlights its capability to handle extended audio. The experiments demonstrated that sinusoidal embeddings consistently outperformed other embedding methods, leading to faster training and better loss values. The pooling mechanism was also shown to be essential, not only for maintaining training and inference speed but also for generating perceptually coherent audio samples over long durations.
In comparisons against prior state-of-the-art models like SaShiMi and Mamba on the SC09 dataset, HarmonicRNN achieved superior results across various metrics, including log-likelihood and perceptual scores like Frechét Inception Distance (FID) and Inception Score (IS). On the Beethoven and YouTubeMix datasets, HarmonicRNN also showed competitive, and in some cases, better, negative log-likelihood scores compared to SaShiMi.
Also Read:
- Flamed-TTS: Advancing Zero-Shot Text-to-Speech with Efficiency and Naturalness
- Improving Secret Audio Recovery with Time-Domain Speech Hiding
The development of HarmonicRNN represents a significant step forward in autoregressive audio generation. By effectively utilizing deep linear RNNs and incorporating thoughtful architectural choices, the model achieves state-of-the-art performance on small-scale datasets, producing high-quality and perceptually coherent long music samples. This work paves the way for future research to scale these approaches to even more demanding audio generation tasks. You can read the full research paper here: Linear RNNs for autoregressive generation of long music samples.
