TLDR: MMMOS is a novel no-reference audio quality assessment system that moves beyond single-score evaluations. It assesses audio across speech, music, and environmental sounds using four orthogonal axes: Production Quality, Production Complexity, Content Enjoyment, and Content Usefulness. By fusing features from specialized pre-trained encoders (WavLM, MuQ, M2D) and employing an ensemble strategy, MMMOS significantly outperforms baselines in accuracy and generalization, achieving top ranks in the AudioMOS 2025 challenge.
Evaluating the quality of audio, whether it’s speech, music, or environmental sounds, is crucial for developing and improving systems that generate, retrieve, or enhance audio. Traditionally, many assessment models have focused solely on speech and provided a single ‘Mean Opinion Score’ (MOS). While simple, this approach often oversimplifies the complex factors that contribute to perceived audio quality and struggles to apply beyond speech.
A new system called MMMOS (Multi-domain Multi-axis Audio Quality Assessment) has been introduced to address these limitations. MMMOS is a no-reference system, meaning it doesn’t need an original, uncorrupted audio sample for comparison. Instead, it estimates audio quality across four distinct and independent aspects:
The Four Pillars of Audio Quality:
- Production Quality (PQ): This measures the technical fidelity, considering factors like clarity, dynamic range, frequency balance, and how sounds are placed in a virtual space.
- Production Complexity (PC): This quantifies how complex an audio scene is, for example, by counting the different sound components present.
- Content Enjoyment (CE): This captures the subjective appeal of the audio, including its emotional impact, artistic expression, and the overall listener experience.
- Content Usefulness (CU): This evaluates how suitable the audio is for practical use, such as being reused in content creation.
MMMOS is designed to work across various audio domains, including speech, music, and environmental sounds, making it far more versatile than previous speech-centric models.
How MMMOS Works:
The system leverages the power of pre-trained AI models specialized in different audio types. It takes an audio waveform and processes it in parallel through three ‘frozen’ encoders:
- WavLM Base+: For speech analysis.
- MuQ: For music analysis.
- M2D: For general audio analysis.
These encoders extract detailed ‘frame-level embeddings’ (numerical representations of the audio at very short time intervals). These representations are then combined and fed into different ‘aggregation modules’ which process the features and predict scores for each of the four quality axes. The system also explores various ‘loss functions’ during training, which are mathematical methods used to guide the model in learning to make accurate predictions.
A key strategy for MMMOS’s success is ‘ensembling,’ where the predictions from multiple top-performing models are combined. This helps to reduce prediction variability and makes the overall quality estimates more stable and reliable.
Also Read:
- Advancing Speech Quality Assessment with a Mixture of Experts Model
- Real-World Audio Fingerprinting: A New Approach to Robust Song Recognition
Performance and Key Findings:
MMMOS was put to the test in the AudioMOS Challenge 2025, Track 2, which evaluates audio samples from text-to-speech (TTS), text-to-audio (TTA), and text-to-music (TTM) systems. The results were impressive:
- MMMOS achieved first place in 6 out of 8 metrics for Production Complexity.
- It ranked among the top three in 17 out of 32 overall challenge metrics.
- Compared to the official baseline, MMMOS showed a significant 20–30% reduction in prediction error (Mean Squared Error) and a 4–5% increase in Kendall’s τ, a metric for ordinal alignment, across all four perceptual axes.
The research also highlighted that combining encoders from different domains (speech, music, general audio) significantly improves the model’s ability to generalize and make robust quality predictions. Furthermore, carefully selecting and combining models through ensembling proved to be an effective way to balance predictive performance with computational efficiency.
This work represents a significant step forward in automatic audio quality assessment, offering a more nuanced and comprehensive understanding of audio quality across diverse soundscapes. For more technical details, you can refer to the full research paper: MMMOS: Multi-domain Multi-axis Audio Quality Assessment.
