TLDR: A new research paper introduces the Speaker Characteristics DeepFake (SCDF) dataset, a novel resource designed to systematically evaluate demographic biases in deepfake speech detection. The dataset, comprising over 237,000 utterances across five languages and a wide age range, reveals that existing deepfake detectors exhibit significant performance disparities based on speaker sex, language, age, and the type of synthesizer used. These findings highlight the critical need for bias-aware development in deepfake detection systems to align with ethical and regulatory standards.
As artificial intelligence continues to advance, the creation of highly realistic deepfake audio has become increasingly sophisticated. While much attention has been given to detecting these synthetic voices, a crucial aspect has remained largely unexplored: bias and fairness in deepfake speech detection systems. This oversight is particularly concerning given the growing regulatory focus on fair AI, such as the European Union’s AI Act, which imposes strict requirements on biometric systems, including deepfake detectors.
To address this critical gap, researchers have introduced a groundbreaking new resource: the Speaker Characteristics DeepFake (SCDF) dataset. This novel dataset is richly annotated, specifically designed to enable a systematic evaluation of demographic biases in deepfake speech detection. It provides a much-needed foundation for understanding how factors like a speaker’s sex, language, and age can influence the performance of deepfake detectors.
The SCDF dataset is a substantial collection, featuring over 237,000 utterances, which amounts to more than 500 hours of speech. It boasts a balanced representation of both male and female speakers across five different languages: Czech, French, English, German, and Spanish. Furthermore, it covers a wide age range, allowing for comprehensive analysis of age-related biases. The dataset was meticulously constructed using the VoxPopuli speech corpus, which provides diverse genuine speech data from European Parliament event recordings. This allowed researchers to gather detailed metadata, including speaker sex, language, and age at the time of recording.
To create the deepfake portion of the dataset, the researchers employed four state-of-the-art speech synthesizers: XTTSv2, F5-TTS, OpenVoice v2, and DDDM-VC. These tools represent both Text-to-Speech (TTS) and Voice Conversion (VC) approaches, ensuring a robust and varied set of synthetic audio for testing.
Also Read:
- Advancing Speech Diarization and Separation Without Prior Enrollments
- AI Agents Are Getting Smarter at Scam Calls, Bypassing Current Defenses
Key Findings on Detector Biases
The experiments conducted using the SCDF dataset revealed significant performance disparities in several state-of-the-art deepfake speech detectors. These findings underscore the urgent need for bias-aware development in this field:
- Sex-Related Bias: The analysis showed a clear bias, with detectors generally performing better (exhibiting lower error rates) for female speakers compared to male speakers. This suggests that deepfake audio of male voices might be harder to accurately identify as fake.
- Language-Related Bias: Performance varied significantly across languages. Interestingly, German deepfakes proved to be the most challenging for the detectors to identify, indicating a potential vulnerability or area for improvement in multilingual detection systems.
- Age-Related Bias: A concerning trend emerged regarding speaker age. The detection performance generally declined with increasing speaker age, meaning that spoofed speech from older speakers was less likely to be correctly identified. The 70+ age group presented the most significant challenge for the examined detectors.
- Synthesizer-Type Bias: As expected, deepfakes created by Text-to-Speech (TTS) models were more challenging for detectors to identify compared to those generated by Voice Conversion (VC) tools. This highlights the need for detectors to be more robust against high-quality TTS-generated deepfakes.
The introduction of the SCDF dataset marks a crucial step towards building more robust, fair, and non-discriminatory deepfake detection systems. By providing a richly annotated resource that allows for systematic bias evaluation, this work directly supports ethical AI principles and helps meet the legal requirements for non-discriminatory biometric systems. The full research paper can be accessed for more detailed information: SCDF: A Speaker Characteristics DeepFake Speech Dataset for Bias Analysis.
