TLDR: This paper introduces a systematic benchmark for evaluating open-source Multimodal Large Language Models (MLLMs) on standard face recognition datasets. It finds that while MLLMs capture semantic cues, they currently underperform specialized face recognition models in high-precision zero-shot scenarios. The study highlights that fine-tuning MLLMs with domain-specific data can improve performance, but a significant gap remains, providing a foundation for future research to develop more accurate MLLM-based face recognition systems.
Multimodal Large Language Models (MLLMs) have made significant strides in understanding both visual and linguistic information, excelling in tasks like image captioning and visual question answering. These powerful models, such as Flamingo, QwenVL, and GPT-4o, combine visual encoders with large language models, allowing them to interpret perceptual inputs and generate contextually relevant text. They represent a new generation of foundation models capable of general-purpose image processing without extensive task-specific training.
However, despite their broad capabilities, the application and performance of MLLMs in the specialized field of face recognition have remained largely unexplored, particularly concerning open-source models. Traditional face recognition systems have well-established benchmarks and protocols, and it’s crucial to understand how MLLMs measure up against these dedicated systems.
Benchmarking MLLMs for Face Recognition
A recent research paper titled “BENCHMARKING MULTIMODAL LARGE LANGUAGE MODELS FOR FACE RECOGNITION” by Hatef Otroshi Shahreza and S´ebastien Marcel from Idiap Research Institute, Switzerland, addresses this gap. The authors conducted a systematic benchmark of state-of-the-art open-source MLLMs to evaluate their effectiveness in face recognition tasks. Their goal was to compare MLLMs with existing, specialized face recognition models on standard datasets using consistent evaluation protocols.
The benchmark focused on a face verification task: given two face images, the MLLM was prompted to answer “yes” or “no” to the question, “Are these two images of the same person?”. This straightforward approach aligns with how traditional face recognition models are typically evaluated.
Datasets Used
The study utilized several widely recognized face recognition datasets to ensure a comprehensive evaluation:
-
Labeled Faces in the Wild (LFW): A foundational dataset for unconstrained face verification, featuring diverse real-world conditions.
-
Cross-Age LFW (CALFW): Challenges models with image pairs of the same individual at different ages, testing robustness to aging effects.
-
Cross-Pose LFW (CPLFW): Focuses on variations in facial pose, evaluating how well systems handle extreme viewpoint changes.
-
Celebrities in Frontal-Profile (CFP): Designed to test recognition across frontal and profile views, including frontal-to-frontal and frontal-to-profile matching.
-
AgeDB-30: A benchmark specifically for age-related variations, using a 30-year age gap protocol.
-
Racial Faces in-the-Wild (RFW): Evaluates bias and fairness across different demographic groups (Caucasian, Asian, Indian, African).
Key Findings and Performance Insights
The experimental results revealed several important insights. While MLLMs demonstrate an ability to capture rich semantic cues useful for face-related tasks, they generally lag behind specialized face recognition models in high-precision recognition scenarios, especially in zero-shot applications. For instance, top-performing MLLMs like Qwen2-VL-7B-Instruct achieved an average accuracy of around 81.10% across the datasets, whereas specialized models like IResNet-50 (MS1MV2) reached an impressive 97.31% average accuracy.
The study also observed that increasing the size of an MLLM can improve performance, but this improvement tends to saturate. A notable finding was the impact of fine-tuning: models like FaceLLM-8B, which is based on InternVL3 and specifically fine-tuned for face understanding, showed improved performance compared to its base model. This suggests that incorporating domain-specific data during training can significantly enhance MLLMs’ capabilities for face recognition.
Furthermore, when evaluating performance across different demographic groups using the RFW dataset, a significant gap persisted between MLLMs and traditional face recognition models. While MLLMs showed varying performance across groups, specialized models maintained consistently high accuracy, highlighting the need for MLLMs to improve fairness and robustness across diverse populations.
Also Read:
- Navigating Document Intelligence with Large Language Models: A Comprehensive Overview
- Unmasking Cultural Differences: How Blur and Background Influence AI’s Facial Expression Recognition
Conclusion and Future Directions
The research concludes that while MLLMs possess considerable potential in various applications, their training on general-purpose datasets often leads to a lack of task-specific precision required for accurate face recognition. They can describe general appearance or basic demographic attributes but struggle with the fine-grained details necessary for identity verification. This benchmark provides a crucial foundation for advancing MLLM-based face recognition, offering valuable insights for designing next-generation models with higher accuracy and better generalization capabilities. Researchers can access the source code of this benchmark to further their studies. For more details, you can read the full research paper here.
