TLDR: A new framework, NLKI, significantly improves small Vision-Language Models (sVLMs) in commonsense visual question answering by integrating natural language facts and LLM-generated explanations. It boosts accuracy by up to 13% on challenging datasets, enabling sVLMs to rival or surpass larger generative models, especially when combined with noise-robust training techniques to handle prevalent label noise.
Small Vision-Language Models (sVLMs) like ViLT, VisualBERT, and FLAVA often struggle with commonsense visual question answering (VQA) tasks. This is because these models frequently lack the crucial external knowledge needed to understand context that isn’t directly visible in an image or explicitly stated in a question. Unlike their larger, more resource-intensive counterparts, sVLMs have historically lagged in tasks requiring a deeper, more human-like understanding of the world.
To address this challenge, researchers have introduced NLKI, a lightweight Natural Language Knowledge Integration framework. NLKI is an innovative, end-to-end system designed to significantly enhance the commonsense reasoning abilities of these smaller AI models. The framework operates by carefully integrating external knowledge into the sVLM’s processing pipeline, allowing them to make more informed and accurate predictions.
How NLKI Works
The NLKI framework is built on three core components:
First, it retrieves relevant natural language facts from commonsense knowledge bases. This is achieved using a fine-tuned ColBERTv2 model, which is optimized to find precise and contextually relevant information for a given image-question pair. This step ensures that the model has access to a broad spectrum of everyday knowledge, from object properties to social conventions.
Second, NLKI leverages a Large Language Model (LLM), specifically Llama-3.1-8B, to craft natural language explanations. These explanations are not just raw facts; they are carefully generated narratives that synthesize visual context (from dense and region captions, and detected objects) with the retrieved facts. This process is crucial for cutting down on ‘hallucinations’—instances where the AI generates incorrect or unsupported details—and for providing coherent, actionable insights to the sVLM.
Finally, both the retrieved facts and the LLM-generated explanations are fed into the sVLMs. This dual input mechanism allows the sVLMs to benefit from both direct factual knowledge and richer, more contextualized reasoning, enabling them to better understand and answer complex commonsense questions.
Also Read:
- Uncovering a Factual Recall Gap in Vision Language Models
- CLARIFY: A New AI Framework for Accurate and Efficient Skin Condition Diagnosis
Key Achievements and Insights
The impact of NLKI on sVLMs is substantial. The framework has been shown to boost end-to-end answer accuracy by up to 7% across various datasets, and by an impressive 13.6% on the AOKVQA dataset. This performance uplift allows NLKI-equipped sVLMs, which typically have around 240 million parameters, to match or even exceed the performance of much larger, medium-sized generative VLMs (up to 4 billion parameters) like Qwen-2 VL-2B and SmolVLM-2.5B.
A critical finding was the importance of ‘Type-5’ explanations. These explanations, which combine dense captions, region captions, detected objects, retrieved facts, and the original question, proved most effective in capturing visual context and reducing inaccuracies. The research also highlighted that LLM-based commonsense knowledge often outperforms direct retrieval from knowledge bases, as LLMs can generate more targeted and coherent explanations.
Furthermore, the study addressed the pervasive issue of label noise in commonsense VQA datasets (estimated at 10-25% in CRIC and AOKVQA). By integrating noise-robust loss functions, such as Symmetric Cross-Entropy (SCE) and Generalised Cross-Entropy (GCE), NLKI further stabilized and improved the performance of sVLMs, adding another 2.5% to 5.5% in accuracy on noisy datasets. This adaptive approach to loss functions, tailored to the dataset’s noise level, ensures stable improvements without architectural changes or extra inference costs.
Despite introducing additional modules like captioners, an object detector, a retriever, and an explainer, the NLKI framework remains lightweight and deployable. The total pipeline latency for a single image-question pair is manageable, and components can be run concurrently or offline to optimize performance, offering a favorable trade-off between efficiency and significant performance gains.
In conclusion, the NLKI framework demonstrates that by intelligently integrating external commonsense knowledge and employing noise-aware training, small vision-language models can achieve a level of commonsense reasoning that rivals or surpasses much larger, multi-billion-parameter models. This opens new avenues for developing more efficient and capable AI systems for complex visual understanding tasks. You can read the full research paper here: NLKI: A lightweight Natural Language Knowledge Integration Framework for Improving Small VLMs in Commonsense VQA Tasks.
