TLDR: AUDETER is a new, large-scale dataset (over 4,500 hours, 3 million clips) designed to improve deepfake audio detection in real-world, “open-world” scenarios. It features diverse synthetic audio from 21 recent speech synthesis models and 4 human voice corpora. Experiments show that models trained on AUDETER significantly outperform existing methods in generalizing to novel deepfake audio and diverse human voices, reducing error rates by 44.1% to 51.6%.
The rapid advancement of speech generation systems has made it increasingly difficult to distinguish between human speech and synthetic audio. This poses significant challenges for authenticity in various applications, from forensic authentication to social media misinformation detection and voice biometric security. While many deepfake detection methods exist, their effectiveness in real-world environments, often referred to as ‘open-world’ scenarios, remains unreliable. This unreliability stems from a domain shift between training and test samples, caused by the vast diversity of human speech and the fast evolution of speech synthesis technologies.
Current datasets used for training and evaluating deepfake audio detectors often fall short in addressing these real-world challenges. They typically lack the diversity and up-to-date audio samples needed for both real and deepfake categories. To bridge this critical gap, researchers have introduced AUDETER (AUdio DEepfake TEst Range), a new large-scale and highly diverse dataset designed for comprehensive evaluation and robust development of generalized models for deepfake audio detection.
AUDETER is an impressive collection, boasting over 4,500 hours of synthetic audio generated by 11 recent Text-to-Speech (TTS) models and 10 vocoders. This results in a broad range of TTS/vocoder patterns, totaling an astounding 3 million audio clips, making it the largest deepfake audio dataset by scale. The dataset is publicly available on GitHub, encouraging further research and development in the field. You can find more details about the research paper here: AUDETER: A Large-scale Dataset for Deepfake Audio Detection in Open Worlds.
Addressing Open-World Challenges
The core problem AUDETER aims to solve is the ‘open-world’ detection challenge. This means detecting deepfake audio generated by novel speech synthesis systems that were not part of the training data, as well as handling human voices with diverse acoustic features and artifacts. Existing detection methods often treat this as a closed-set binary classification problem, optimized for limited audio patterns, and thus fail to generalize to new patterns encountered in real-world deployment.
Through extensive experiments using AUDETER, the researchers revealed significant limitations of current state-of-the-art (SOTA) methods. These methods, when trained on existing datasets, struggle to generalize to novel deepfake audio samples and exhibit high false positive rates on unseen human voices. This underscores the urgent need for a more comprehensive dataset like AUDETER.
AUDETER’s Impact on Detection Performance
The research demonstrates that models trained on AUDETER achieve highly generalized detection performance. They significantly reduce the detection error rate by 44.1% to 51.6%, achieving an error rate of only 4.17% on diverse cross-domain samples in the popular In-the-Wild dataset. This remarkable improvement paves the way for training generalist deepfake audio detectors that are much more robust in real-world applications.
AUDETER’s design incorporates several key advantages. It includes real audio samples from four diverse corpora (In-the-Wild, Common Voice, People’s Speech, and Multilingual LibriSpeech), capturing comprehensive human speech variability. For each real audio sample, corresponding fake audio is provided, generated by all synthesis systems using matching scripts, allowing for systematic and balanced evaluation. The dataset also includes audio from 21 recent speech synthesis systems, including cutting-edge TTS models and vocoders, ensuring coverage of diverse and up-to-date deepfake speech patterns.
Ensuring Data Quality
To guarantee the quality of the generated audio, the researchers conducted thorough intelligibility and naturalness assessments. Intelligibility was evaluated using automated speech recognition (ASR) models, measuring metrics like Word Error Rate (WER) Similarity. Naturalness was assessed using Mean Opinion Score (MOS) predictions via the NISQA framework. These assessments confirmed that modern TTS models in AUDETER produce high-quality audio, often indistinguishable from human speech, and offer distinctly different and superior intelligibility patterns compared to vocoders.
Also Read:
- Beyond Meaning: A New Approach to Spotting Synthetic Speech
- New AI Method Boosts Deepfake Audio Detection Across Languages
Future Directions
The introduction of AUDETER marks a significant step forward in deepfake audio detection. It serves as a valuable resource for training open-world detectors and promotes a data-centric approach to improving detection performance. The researchers plan to continue developing AUDETER as an ongoing project, recognizing the rapidly evolving nature of speech synthesis systems. Future work includes identifying representative synthesis patterns that can generalize across multiple systems and exploring advanced training methodologies like self-supervised pretraining to further enhance generalization performance.
