TLDR: The research paper “ArtFace: Towards Historical Portrait Face Identification via Model Adaptation” introduces a novel method for identifying sitters in historical portraits. It combines fine-tuned vision-language foundation models (CLIP with LoRA) with adapted conventional face recognition networks (AntelopeV2/IResNet100). By fusing the embeddings from these models, ArtFace significantly improves accuracy over existing state-of-the-art methods, effectively overcoming challenges like domain shift and stylistic variations inherent in historical artworks.
Identifying the subjects, or ‘sitters,’ in historical paintings has long been a complex and often subjective task for art historians. Traditional facial recognition systems, while highly effective with photographs, struggle significantly when faced with the unique challenges of painted portraits. These challenges include vast stylistic variations, the interpretive nature of art, and a general lack of consistent data, all of which create a substantial ‘domain shift’ that hinders accurate identification.
Early attempts to automate this process involved statistical assessments of local and anthropometric facial features. More recently, deep learning, particularly convolutional neural networks (CNNs), has been employed. Researchers have used Siamese CNNs, fine-tuned models like ResNet-100 with augmented datasets, and combined deep learning with handcrafted features to make progress in this difficult area.
Introducing ArtFace: A Hybrid Approach
A new research paper, ArtFace: Towards Historical Portrait Face Identification via Model Adaptation, introduces a novel approach that leverages the power of modern foundation models to significantly improve sitter identification in historical artworks. The core idea is to bridge the gap between photographic and painted domains by adapting and fusing different types of AI models.
The ArtFace method makes three key contributions: First, it fine-tunes a vision-language foundation model called CLIP using a technique called Low-Rank Adaptation (LoRA) on portrait face images. CLIP, trained on a massive dataset of image-text pairs, excels at understanding broad visual concepts and stylistic differences, which is crucial for art. Second, it complements CLIP with embeddings from an adapted conventional face recognition network, specifically AntelopeV2, which is built on the IResNet100 architecture. AntelopeV2 is known for its high accuracy and robustness in typical face recognition tasks. Third, it fuses the embeddings from these adapted models using a simple yet effective normalized concatenation strategy.
How It Works: Model Adaptation and Fusion
Both the CLIP and AntelopeV2 models undergo adaptation to better suit the domain of historical paintings. For CLIP, LoRA is applied, which efficiently fine-tunes the model with fewer trainable parameters and reduced memory usage. For IResNet100, the adaptation is minimal, involving only the fine-tuning of its final linear layer to adjust embeddings for the new domain. After individual normalization, the embeddings from the tuned and untuned IResNet100 models, along with the tuned CLIP model, are concatenated and re-normalized. This fusion process combines the complementary information captured by each model, leading to a more robust representation.
Experimental Validation and Results
The researchers conducted comprehensive experiments on the public Historical Faces dataset, which includes 766 paintings of 210 different sitters. The dataset was carefully split for training, validation, and testing. The CLIP-LoRA model was trained using triplet loss with hard negative mining, a technique that helps the model learn from the most challenging examples. The IResNet100 model was trained similarly, but without LoRA layers.
The results demonstrate notable improvements over current state-of-the-art methods. While tuning IResNet100 alone showed modest gains, the fusion of tuned and untuned versions significantly outperformed either individually. The CLIP-LoRA tuning, especially with hard negative mining, also showed substantial improvements over the base CLIP model. Crucially, the fusion of CLIP-LoRA with IResNet100 models consistently enhanced overall accuracy across various evaluation metrics, particularly in scenarios requiring low false acceptance rates. This indicates that foundation models capture valuable information from portraits that conventional networks might miss.
Also Read:
- Securing Facial Recognition: A Federated Learning Approach for Identifying Known and Unknown Faces
- MobileCLIP2 Advances Efficient Multi-Modal AI for Mobile Devices
Conclusion
The ArtFace research demonstrates that a combination of lightweight tuning of vision-language foundation models and domain-adapted face recognition networks can effectively bridge the domain gap between photographs and paintings. This fusion approach achieves state-of-the-art accuracy in sitter identification, proving that adapting modern AI architectures is feasible and promising for this challenging task. The work opens new avenues for research, including the use of synthetic data generation and heterogeneous domain adaptation techniques to further improve generalization across diverse visual domains in art history.
