TLDR: A study compared open-source Vision-Language Models (VLMs) like Phi-3.5 Vision and CLIP with traditional deep learning models (VGG19, ResNet-50, EfficientNet-B0) for facial emotion recognition on the challenging FER-2013 dataset. It found that traditional models, especially EfficientNet-B0, significantly outperformed VLMs, suggesting VLMs struggle with low-quality, noisy visual data despite a novel image restoration pipeline. The research highlights the need for VLMs to adapt to real-world noisy environments and provides a benchmark for future studies.
Facial Emotion Recognition (FER) is a vital area in artificial intelligence, with wide-ranging applications from improving human-computer interactions to aiding in mental health diagnostics. Traditionally, deep learning models like VGG19, ResNet-50, and EfficientNet-B0 have shown strong capabilities in this field by effectively extracting visual features from faces.
However, real-world scenarios often present challenges such as low-resolution, noisy, and grayscale images, as seen in the widely used FER-2013 dataset. This dataset, containing nearly 36,000 images across seven emotion categories (angry, disgust, sad, happy, neutral, surprise, and fear), is particularly difficult due to its inherent noise, varying lighting conditions, and class imbalances.
Recent advancements in Vision-Language Models (VLMs), such as Phi-3.5 Vision and CLIP, have sparked interest due to their ability to understand both images and text, and their large-scale pretraining. Despite their success in general vision tasks, their performance in FER, especially with noisy and low-resolution data, has remained largely unexplored.
This research addresses this gap by introducing a new approach that combines GFPGAN-based image restoration with FER evaluation. GFPGAN is a powerful tool designed to restore lost facial details and improve the clarity of degraded images, transforming low-resolution grayscale inputs into clearer, higher-resolution color outputs. This preprocessing step aims to enhance the quality of the FER-2013 dataset, allowing models to better detect subtle facial cues.
Comparing the Models
The study systematically compared the performance of traditional deep learning models (VGG19, ResNet-50, EfficientNet-B0) against open-source VLMs (Phi-3.5 Vision, CLIP) on the enhanced FER-2013 dataset. The evaluation focused on key metrics like accuracy, precision, recall, and F1-score, which provide a comprehensive understanding of a model’s ability to classify emotions, especially in datasets with imbalanced classes.
The findings revealed that traditional deep learning models generally outperformed the VLMs. EfficientNet-B0 emerged as the top performer among all tested models, achieving an impressive accuracy of 86.44%. ResNet-50 also showed strong results with 85.72% accuracy. In contrast, VLMs like CLIP and Phi-3.5 Vision achieved accuracies of 64.07% and 51.66% respectively. This highlights a significant limitation of current VLMs: their reliance on structured, high-quality data makes them struggle with the real-world visual variability and noise present in datasets like FER-2013.
Computational Considerations
Beyond performance, the study also provided a detailed analysis of the computational costs involved, covering preprocessing, training, inference, and evaluation phases. While EfficientNet-B0 delivered high accuracy with relatively low computational demands, VLMs like CLIP and Phi-3.5 Vision required more resources for inference and evaluation, even without extensive training. The GFPGAN preprocessing step also added to the overall computational overhead. These insights are crucial for practical deployment, especially in environments with limited computing resources.
Also Read:
- Assessing AI’s Ability to Mimic Low Vision Perception
- Improving Video Quality Assessment with Integer-Only Loss Fine-tuning
Future Outlook
This research underscores the need for adapting VLMs to handle noisy and low-quality visual data more effectively. Future work could explore integrating deep learning models with VLMs to combine robust feature extraction with contextual understanding. Fine-tuning VLMs specifically for challenging datasets like FER-2013, expanding experiments to more diverse datasets, and optimizing computational efficiency are also important directions for advancing facial emotion recognition systems. For more details, you can refer to the full research paper here.
