TLDR: The Hierarchical Deep Fusion Framework (HDFF) is an ensemble deep learning architecture designed for facial forgery detection. It combines four diverse pre-trained models (Swin-MLP, CoAtNet, EfficientNetV2, DaViT) through a multi-stage fine-tuning process. By concatenating features from these specialized models and training a final classifier, HDFF achieved a score of 0.96852, ranking 20th out of 184 teams in the 2024 Global Deepfake Image Detection Challenge, demonstrating an effective strategy for complex image classification tasks.
The rapid advancement of generative artificial intelligence has led to the creation of increasingly realistic deepfakes, posing significant threats to digital security and the authenticity of information. These manipulated facial images and videos, generated through techniques like face swapping and attribute editing, necessitate sophisticated detection systems that can generalize across a wide array of forgery methods.
Addressing this critical need, researchers have introduced the Hierarchical Deep Fusion Framework (HDFF), an innovative ensemble-based deep learning architecture. This framework was specifically designed for high-performance facial forgery detection and was a key participant in the 2024 Global Deepfake Image Detection Challenge, which featured the comprehensive MultiFFDI dataset with over 50 different forgery generation methods.
A Novel Approach to Deepfake Detection
Traditional detection methods often struggle to adapt to new, unseen forgery techniques. The HDFF overcomes this limitation by leveraging the power of ensemble learning, but with a more advanced strategy than simply averaging model outputs. Instead, it fuses features from multiple diverse models, allowing a final classifier to learn the intricate relationships between these different feature representations.
The core of the HDFF integrates four distinct and powerful pre-trained sub-models: Swin-MLP, CoAtNet, EfficientNetV2, and DaViT. These models, chosen for their diverse architectures—ranging from pure transformers and MLPs to hybrid convolution-attention models—are capable of capturing a broad spectrum of forgery artifacts. The framework operates by using each fine-tuned sub-model as a feature extractor. The outputs from their final layers are then concatenated into a single, larger feature vector. This combined vector is fed into a new, fully connected layer, which acts as the ultimate classifier, learning to weigh the features from each sub-model to produce a highly accurate final classification.
A practical constraint of the challenge was a model size limit of 200MB. The HDFF successfully managed its parameters, with a total count of 180.16M, comfortably staying within the allocated budget.
Multi-Stage Training for Optimal Performance
A crucial element of the HDFF’s success is its meticulously designed multi-stage training strategy. This approach ensures that both the individual sub-models and the final fusion layer are optimally trained:
-
Sub-Model Initialization: Each of the four sub-models begins with weights pre-trained on the ImageNet-1K dataset, providing a strong foundation for general feature extraction.
-
Selective Fine-Tuning: Initially, all layers of each sub-model are frozen except for the final classifier layer. This layer is then trained on the MultiFFDI dataset, allowing it to quickly adapt to the new domain without disrupting the learned feature hierarchies.
-
Comprehensive Fine-Tuning: After the classifier head adapts, all layers of each sub-model are unfrozen and training continues on the MultiFFDI dataset with a smaller learning rate. This end-to-end fine-tuning enables the entire network to specialize in detecting subtle forgery artifacts.
-
Grand Model Training: Once individual sub-models are optimized, their weights are frozen, and only the newly added final fully connected fusion layer is trained. This efficient step focuses solely on learning the best way to combine the powerful features extracted by the sub-models.
Data preprocessing and augmentation also played a vital role in enhancing model robustness. The framework utilized AutoAugment, which applies a learned sequence of augmentation policies, including transformations like ‘Invert’, ‘Rotate’, and ‘Brightness’, to prevent overfitting and improve generalization.
Also Read:
- MetaSeal: A New Approach to Image Attribution Security
- Advancing Person Re-Identification Across Visible and Infrared Modalities Without Manual Labels
Impressive Results and Key Insights
The Hierarchical Deep Fusion Framework achieved a final score of 0.96852 on the competition’s private leaderboard, securing the 20th position out of 184 participating teams. This strong performance validates the effectiveness of the multi-stage training and hierarchical fusion approach for complex multi-dimensional facial forgery detection.
During development, several critical observations emerged. The use of a CosineAnnealingLR scheduler, which smoothly adjusts the learning rate, provided a significant performance boost compared to a standard StepLR scheduler. Furthermore, the necessity of full-network fine-tuning was underscored; initial experiments with only training the final layer severely limited model potential, highlighting that for complex, domain-specific tasks like forgery detection, adapting the entire feature extraction backbone is often essential for optimal results.
The HDFF represents a robust method for combating the proliferation of deepfakes. While the framework achieved a high rank, future work could explore multi-modal features, such as predicting and analyzing depth maps from facial images for inconsistencies, to further enhance detection capabilities. For more technical details, you can read the full research paper here.
