TLDR: DeMul (Description-free Multi-prompt Learning) is a new method that improves Vision Language Models by directly distilling knowledge from Large Language Models into prompts, bypassing unreliable text descriptions. It uses learnable, weighted prompts in the LLM’s embedding space for better semantic capture and introduces dynamic prompt weighting. This approach consistently outperforms existing methods in image classification across 11 datasets, simplifying prompt learning and enhancing accuracy.
In the evolving landscape of artificial intelligence, Vision Language Models (VLMs) have shown remarkable ability to adapt to new tasks without needing extensive, specially labeled datasets. These models, which understand both images and text, often rely on “prompt learning” – essentially, guiding the model with specific text phrases to help it understand what to look for.
Traditionally, these prompts would sometimes incorporate descriptions generated by Large Language Models (LLMs) like GPT. For example, to classify a “Golden Retriever,” an LLM might generate descriptions of its features. However, this approach has faced challenges: LLM-generated descriptions can be highly variable, sometimes unreliable, and might even contain biases or ambiguous language. Imagine asking an LLM to describe a Golden Retriever, and it gives you features that are hard to verify visually or are too general, like “often has a friendly demeanor.”
A new method, called Description-free Multi-prompt Learning (DeMul), offers an innovative solution to these issues. Instead of relying on LLMs to generate explicit text descriptions, DeMul directly distills knowledge from the LLM into the prompts themselves. This means the prompts can capture richer, more nuanced meanings without being tied to discrete, pre-defined text templates. Think of it as directly infusing the “essence” of what an LLM understands about a concept into the prompt, rather than asking it to write a sentence about it.
DeMul achieves this by mapping learnable prompts into the LLM’s embedding space – a high-dimensional space where words and concepts are represented as vectors. By optimizing these prompt vectors to align with the LLM’s understanding of class names, DeMul ensures that the prompts absorb meaningful semantics directly. The researchers used OpenAI’s GPT-based embedding models, specifically the ‘text-embedding-3-large’ model, known for its reliable performance in text similarity tasks.
Furthermore, DeMul introduces a “prompt weighting” mechanism. In a multi-prompt setting, where a class might have several prompts, not all prompts contribute equally to the classification task. This weighting system dynamically adjusts the importance of each prompt during training, allowing the model to emphasize more informative prompts and reduce the impact of less relevant ones. This is particularly useful because the optimal number and importance of prompts can vary greatly between different classes.
The effectiveness of DeMul was rigorously tested across 11 diverse image recognition datasets, including ImageNet, Stanford Cars, and Oxford Flowers. Using CLIP, a widely researched Vision Language Model, as a baseline, DeMul consistently demonstrated superior performance. For instance, it achieved a significant increase in accuracy compared to the CLIP baseline, and even outperformed other state-of-the-art prompt learning methods like GalLoP. The results highlight that directly training for classification utility, rather than embedding descriptive semantics, yields more meaningful benefits.
Also Read:
- Enhancing Multi-modal Reasoning in AI Models with Rationale-Enhanced Decoding
- Beyond Single Scenes: MVL-Loc Enhances Camera Positioning with Vision-Language Models
This novel approach simplifies the prompt learning process and enhances the accuracy of image classification tasks, marking a significant advancement in how pre-trained models are utilized. For more technical details, you can read the full research paper here.
