TLDR: Vision-and-Language (VL) training helps language models (LMs) better apply their existing taxonomic knowledge, such as understanding “a cat is an animal,” even in text-only tasks. Researchers found that VL models don’t fundamentally change their core taxonomic understanding compared to text-only LMs. Instead, the visual training improves how these models use this knowledge in specific contexts, especially when visual similarities between concepts are relevant.
A recent study delves into a fascinating question: Does adding vision-and-language (VL) training to language models (LMs) fundamentally change how they understand linguistic concepts, particularly their taxonomic organization (like knowing a ‘cat’ is an ‘animal’)? The findings suggest that while VL training significantly improves how these models apply their knowledge, it doesn’t necessarily alter the core knowledge itself.
The research, titled “Vision-and-Language Training Helps Deploy Taxonomic Knowledge but Does Not Fundamentally Alter It,” was conducted by a team of researchers including Yulu Qin, Dheeraj Varghese, Adam Dahlgren Lindström, Lucia Donatelli, Kanishka Misra, and Najoung Kim. Their work sheds light on the subtle yet impactful ways multimodal training influences AI models.
Historically, studies comparing text-only LMs with their VL-trained counterparts have shown inconsistent or marginal differences in their linguistic capabilities. This paper hypothesized that a domain where VL training could have a significant effect is lexical-conceptual knowledge, specifically its taxonomic structure.
To investigate this, the researchers developed a new dataset called TaxonomiGQA. This dataset is a text-only version of the popular visual-question answering (VQA) dataset GQA, synthetically augmented to require taxonomic understanding. They converted scene graphs into textual descriptions and created questions that involved substituting objects with their hypernyms (broader categories) from the WordNet hierarchy, along with negative samples. This allowed for a systematic comparison of how LMs and VLMs handle questions requiring taxonomic reasoning, even when presented purely in text.
The study compared seven widely used VLM-LM minimal pairs. Surprisingly, most VLMs consistently outperformed their LM counterparts on this text-only question-answering task. This initial observation raised two possibilities: either VL training fundamentally alters the taxonomic knowledge, or it improves the model’s ability to deploy its existing knowledge.
To test the first hypothesis, the team introduced TAXOMPS (Taxonomic Minimal Pairs), a dataset designed to directly elicit taxonomic judgments (e.g., “Is it true that a cat is an animal?”). The results showed that most VLM-LM pairs performed quite similarly on this task, suggesting that VL training does not generally change the basic taxonomic membership judgments of a language model. Further analysis of lexical representations, including hierarchical structure and embedding similarities, also indicated that the underlying taxonomic knowledge in VLMs and LMs remained largely shared and fundamentally similar.
This led the researchers to focus on the second hypothesis: VLMs are better at deploying taxonomic knowledge. Through analyses of contextualized lexical representations and Principal Component Analysis (PCA) of question representations, they found compelling evidence. VLMs showed stronger connections between their representations and correct answers in task contexts requiring taxonomic knowledge. Additionally, the representations of questions containing taxonomic relations were more clearly separable from those without such relations in VLMs compared to LMs.
So, why does vision training help? The study offers a preliminary investigation, suggesting that visual similarity between members of concepts in a hypernym-hyponym relation might be a key factor. For instance, the visual similarity between different types of ‘equine’ (like horses) is higher than that between different ‘vertebrates’ (like fish and mammals). The results indicated that VLMs’ success on TaxonomiGQA could be predicted by the visual similarity between concepts in a taxonomic relation, and this prediction strength was modulated by the visual cohesion of the hypernym category.
Also Read:
- Unpacking How Large Language Models Grasp Story Analogies
- VisionThink: Smarter Image Processing for AI Models
In conclusion, the research highlights that while vision-and-language training doesn’t fundamentally restructure a language model’s inherent taxonomic knowledge, it significantly enhances its ability to apply this knowledge effectively in relevant tasks, even when the task is purely linguistic. This improvement appears to be linked to the model’s capacity to leverage visual similarities between concepts. For more details, you can read the full research paper here.
