TLDR: This research introduces a new speech quality assessment model based on a Mixture of Experts (MoE) architecture, enhanced with multi-task learning and extensive data augmentation. The model significantly improves system-level Mean Opinion Score (MOS) prediction accuracy by adaptively processing diverse audio data and learning from different speech synthesis technologies. While excelling at system-level evaluation, the model shows limited improvement in utterance-level prediction, highlighting the complexity of fine-grained quality assessment and suggesting areas for future research like rater adaptation and fine-grained feature extraction.
Automatic speech quality assessment is vital for developing advanced speech synthesis systems. However, existing models often struggle with consistent performance across different levels of detail in their predictions, and they face challenges when applied to new or unfamiliar speech systems.
Researchers have introduced a new approach to improve the Mean Opinion Score (MOS) prediction, which is the gold standard for evaluating human perception of speech quality. This novel system is built upon self-supervised learning speech models and incorporates a specialized Mixture of Experts (MoE) classification head. To enhance its capabilities, the system also uses synthetic data from various commercial speech generation models for data augmentation.
The proposed method introduces three key innovations. Firstly, it features a dedicated MoE classification head that uses a gating mechanism to adaptively process diverse audio data. This allows the model to learn from different types of speech characteristics more effectively. Secondly, a multi-task learning strategy is employed, combining the primary task of MOS prediction with an auxiliary task of classifying the synthetic model used to generate the speech. This helps the model understand the “fingerprint” features left by different synthesis technologies. Lastly, the system benefits from systematic data augmentation, effectively doubling the training dataset size by incorporating data from four state-of-the-art commercial synthesis models: CosyVoice-300M, CosyVoice2-0.5B, FireRedTTS, and f5tts 24khz.
The dataset used in this research initially comprised 400 audio samples from eight different speech synthesis models. Through the data augmentation process, 400 new samples were generated, expanding the total dataset to 800 samples. This not only increased the number of samples but also broadened the coverage of speech synthesis models from 8 to 12, enhancing technological diversity.
The overall architecture of the framework is end-to-end, consisting of a feature extraction backbone network, a feature fusion module, a hybrid expert classification head, and a multi-task output layer. The hybrid expert system design involves multiple independent expert networks, each a multi-layer fully connected network with distinct parameters, designed to learn deep feature representations of specific audio types. A gating network dynamically assigns weights to these experts based on input features, allowing the model to adaptively select the most appropriate processing path.
Training of the model follows a three-stage progressive strategy. The first stage involves pre-training on large-scale open-source multi-model datasets for classification tasks. The second stage introduces the target MOS dataset for joint training of both MOS prediction and model classification. The final stage fine-tunes the model specifically on the target MOS dataset to optimize MOS prediction performance.
In terms of performance, the new system achieved substantial improvements in system-level MOS prediction, significantly reducing the mean square error (MSE) compared to baseline models. This indicates a strong improvement in absolute prediction accuracy. However, the improvements in correlation metrics like LCC, SRCC, and KTAU were limited, suggesting the model primarily learned to improve absolute accuracy rather than enhancing relative quality ranking capabilities.
Also Read:
- Unlocking Piano Expression: A New Model for Continuous Sustain Pedal Analysis
- Advancing Singing Voice Synthesis for Bollywood Hindi with LAPS-Diff
A key finding of the research is that while the system excels in system-level evaluation, its performance improvements in utterance-level MOS prediction remain limited. This is attributed to the fundamental differences between the two assessment levels: system-level evaluation focuses on overall technical differences, while utterance-level evaluation requires capturing fine-grained quality variations and is more susceptible to individual listener biases. The paper suggests future strategies to enhance utterance-level prediction, including rater adaptation, fine-grained feature extraction, and personalized prediction. For more details, you can refer to the original research paper here.
