TLDR: This research introduces an efficient method for “Audio Coding for Machines” (ACoM) that repurposes parts of trained machine learning models to compress audio into ultra-low bitrate discrete features. By integrating residual vector quantization (RVQ) and task-specific loss guidance, the method achieves significant compression (under 200 bps) with minimal performance loss on downstream tasks like automatic speech recognition and audio classification, making it ideal for split computing architectures.
A new research paper introduces an innovative approach to audio coding, specifically designed to optimize performance for machine learning tasks rather than human perception. This method, termed Audio Coding for Machines (ACoM), aims to achieve highly efficient compression while ensuring that the compressed audio remains effective for downstream machine learning applications like automatic speech recognition (ASR) and audio classification (AC).
Traditionally, audio codecs are built to preserve the highest possible fidelity of sound for human listeners, which often results in large data sizes and higher bitrates. However, for machines, many of these fine-grained audio nuances are redundant and inefficient. The ACoM paradigm shifts this focus, prioritizing the utility of discrete audio features for machines, even if it means disregarding aspects important for human perception.
The core of this new method involves repurposing the feature transformation layers of an already trained speech or audio machine learning model to act as an audio tokenizer. Instead of relying on a separate, general-purpose audio codec, the early layers of a machine learning model are used to convert raw audio input into a compact, discrete feature representation. These discrete features, or ‘codes,’ are then transmitted to a cloud server where the remaining layers of the machine learning model process them to perform the intended task. This integrated approach eliminates the need for a separate decoding step, streamlining the entire pipeline.
The researchers incorporated Residual Vector Quantization (RVQ) directly into the training process of the machine learning model. By guiding this quantization with task-specific loss functions alongside RVQ losses, the system learns to generate ultra-low bitrate codes (less than 200 bits per second) with minimal degradation in the downstream model’s performance. This results in a tokenizer that is adaptable to various bitrates and model sizes, offering flexible deployment options.
This novel approach brings several significant advantages. Firstly, it drastically reduces the bitrate, leading to faster and more efficient data transmission. Secondly, it removes the computational overhead associated with running a standalone audio codec, as the encoding function is seamlessly integrated into the existing machine learning model. Lastly, the method is highly versatile and can be applied to various speech and audio tasks, accommodating different resource constraints on edge devices by allowing parts of the model to be offloaded for on-device processing.
The efficacy of the method was demonstrated through experiments on two popular tasks: Automatic Speech Recognition (ASR) using the LibriSpeech dataset and Audio Classification (AC) using the UrbanSound8k dataset. For ASR, the proposed method achieved Word Error Rates (WERs) remarkably close to a continuous baseline model, even at significantly lower bitrates. For instance, at 500 bps, it outperformed conventional codec approaches, especially on noisy inputs. It even achieved ultra-low bitrates around 130 bps with an acceptable trade-off in performance.
Similarly, for audio classification, the method showed excellent results, achieving a classification accuracy of 80.2% at a raw bitrate of 200 bps, which was slightly better than the continuous baseline’s 79.7%. This highlights the method’s capability to maintain or even enhance performance while drastically compressing the data. The research indicates a clear trade-off between the choice of quantization layer, bitrates, and classification accuracies, offering design flexibility based on computational resources.
Also Read:
- QTTS: Advancing Speech Synthesis with Multi-Codebook Audio Compression
- JSQA: Enhancing Speech Quality Assessment with Perceptually-Guided AI
In summary, this research introduces a powerful task-specific quantization strategy that significantly improves the computational efficiency of encoder-decoder tasks under the ACoM paradigm. By optimizing for machine ‘understanding’ rather than human perception, it enables ultra-low bitrates and offers substantial benefits for split computing architectures. You can find more details about this research in the full paper available at this link.
