TLDR: A new study introduces a comprehensive framework and dataset to evaluate the color vision capabilities of Large Vision-Language Models (LVLMs). It defines specific testing tasks (easy and hard), analyzes model performance across various categories, identifies common error types, and proposes fine-tuning strategies that significantly improve color perception accuracy in these AI models. The research highlights the varying strengths of different LVLMs in color recognition and emphasizes the need for human-aligned evaluation metrics.
Large Vision-Language Models (LVLMs) are becoming increasingly common in various critical applications, from autonomous driving to pharmaceutical research. Their ability to accurately perceive and interpret colors is vital for safety and reliability in these real-world scenarios. However, despite their widespread adoption, the color vision capabilities of these advanced AI models have not been thoroughly investigated until now.
Traditional methods for assessing color vision, such as the Ishihara Color Vision Test used for humans, have not been widely applied to LVLMs. Previous research on these models primarily focused on general capabilities like image captioning and object recognition, often overlooking the nuances of visual perception, especially color vision. This left a significant gap in understanding how well these models truly interpret and process color information, particularly in distinguishing primary colors like red, green, and blue, and quantifying any color vision deficits.
To address this crucial gap, a new study introduces a dedicated color vision testing framework for LVLMs. The researchers defined a specific color vision testing task and constructed a comprehensive dataset designed to evaluate these models. This dataset includes 5,450 color vision test images, covering multiple categories of questions and tasks with varying difficulty levels. The categories include numbers, animals, shapes, letters/Chinese characters, and objects, ensuring a broad spectrum of testing scenarios.
The study developed two main tasks: Color Vision Test Easy (CVTE) and Color Vision Test Hard (CVTH). The CVTE provides additional clues about the content category (e.g., numbers, animals) to the model, making the task simpler. In contrast, the CVTH presents prompts without such clues, posing a greater challenge. The dataset itself was meticulously designed using the Daltonlens toolkit, a recognized tool in color vision testing, to generate diverse images with various primary and saturated color schemes, while also preventing potential data leakage.
To evaluate the models, a combination of metrics was used, including machine metrics (Meteor score), model scores (determined by GPT-4 acting as a judge), and human evaluation scores. Interestingly, the analysis revealed that the scores generated by the large language model (GPT-4) aligned more closely with human evaluations compared to traditional machine metrics, highlighting the importance of human-like evaluation criteria for such tasks.
Experiments were conducted on a range of state-of-the-art LVLMs, including both closed-source models like GPT-4o, Gemini-2.0-Pro, and GLM4V-Plus, and open-source models such as JanusPro-7B and LLaVANext-7B. The results showed varying capabilities among the models. For the easier CVTE task, JanusPro-7B achieved the highest average score, slightly outperforming GPT-4o. However, for the more challenging CVTH task, GPT-4o demonstrated superior performance. This suggests that some models might excel with additional context, while others perform better in more straightforward assessments.
Further analysis revealed that different models have specialized strengths across different domains. For instance, GPT-4o performed strongly in number and object recognition, while JanusPro-7B showed superior capabilities in recognizing animals, letters, and shapes. The study also explored the types of errors made by these models, categorizing them into issues like incorrect category understanding, unidentifiable content, complete recognition errors, partial recognition errors, and stochastic fallback on uncertainty.
Also Read:
- Unpacking Vision-Language Models: Where Visual Understanding Falls Short
- COLIBRI: A New Model for Understanding How Humans Perceive Color
Crucially, the research also proposed and tested fine-tuning strategies to enhance model performance. Applying LoRA fine-tuning to LLaVA1.5-7B resulted in a significant improvement in accuracy for both CVTE and CVTH tasks, demonstrating the potential for substantial performance gains through targeted adjustments. This research provides a foundational step towards better understanding and improving the color vision capabilities of large vision-language models, which is essential for their safe and reliable deployment in real-world applications. You can read the full research paper here: Assessing Color Vision Test in Large Vision-language Models.
