TLDR: The paper introduces PMTFR, a new framework for Composed Image Retrieval (CIR) that combines a reference image and text to find target images. It features a Pyramid Patcher for multi-granular visual understanding and a Training-Free Refinement method that injects “reasoning-augmented representations” into Large Vision-Language Models (LVLMs). This approach significantly improves retrieval accuracy on supervised CIR tasks like Fashion-IQ and CIRR, offering better performance and efficiency without needing to train additional ranking models.
Composed Image Retrieval (CIR) is a challenging task that goes beyond traditional image search. Instead of just an image, CIR requires understanding a reference image along with a textual instruction describing modifications to find relevant target images. Imagine wanting to find a ‘blue dress with short sleeves’ when you only have a picture of a ‘red dress with long sleeves’. This is where CIR comes in, but it’s complex due to the need to understand both visual and textual information simultaneously.
Existing methods often struggle with this complexity. Some use a two-stage approach, which typically involves training an additional ranking model, adding to computational costs. While Chain-of-Thought (CoT) techniques have been successful in language models for reducing training expenses, their application in supervised CIR has been limited, often requiring visual information to be compressed into text or relying on intricate prompt designs. Furthermore, CoT has primarily been used for zero-shot CIR, finding it difficult to achieve satisfactory results in supervised settings with well-trained models.
To address these challenges, researchers have proposed a novel framework called the Pyramid Matching Model with Training-Free Refinement (PMTFR). This framework aims to enhance supervised CIR by improving how models understand visual information and by refining retrieval results without additional training.
The Pyramid Matching Model
At the core of PMTFR is the Pyramid Matching Model. This model is designed to learn a general representation of multimodal queries (reference image + modified text) and target images. It uses a simple yet effective module called the Pyramid Patcher. Inspired by multi-scale techniques in visual detection, the Pyramid Patcher helps the model understand visual information at different levels of detail, from broad backgrounds to fine-grained features. Instead of just processing an image as a single set of patches, it divides the image into multiple tokens with varying visual receptive fields. This significantly boosts the model’s visual understanding without adding excessive computational overhead.
Training-Free Refinement with Reasoning-Augmented Representation
One of the most innovative aspects of PMTFR is its Training-Free Refinement paradigm. In supervised CIR, models are trained on specific datasets. Some multi-stage methods try to improve results by training a separate ranking model, but this is resource-intensive. PMTFR offers a solution to refine retrieval results without this extra training.
The key here is ‘Reasoning-Augmented Representation’ (RAug-Rep). Inspired by representation engineering, which involves extracting representations from large language models that reflect specific capabilities, PMTFR extracts these representations from Chain-of-Thought data. Instead of relying on explicit textual reasoning paths (which can be computationally expensive), these RAug-Reps are injected into the intermediate layers of a pre-trained Large Vision-Language Model (LVLM) during the inference phase. This injection subtly guides the model, allowing it to obtain refined retrieval scores as if it were performing explicit reasoning, but without the computational cost. It’s like finding a ‘key’ that unlocks a specific capability of the model, leading to performance improvements by simply inserting it.
Performance and Efficiency
Extensive experiments conducted on popular CIR benchmarks like Fashion-IQ and CIRR demonstrate that PMTFR surpasses state-of-the-art methods in supervised CIR tasks. For instance, on the Fashion-IQ dataset, PMTFR showed an average improvement of 1.42% over CIR-LVLM and a significant 5.44% over the two-stage Re-ranking method, all without the need for additional ranking model training. Similarly, on the CIRR dataset, it outperformed CIR-LVLM by 1.22% on average and Re-ranking by 1.76%.
The Training-Free Refinement significantly reduces time consumption compared to training a separate ranking model, while still achieving comparable or better results. This highlights PMTFR’s efficiency and effectiveness.
Also Read:
- Enhancing MLLM Accuracy: A New Method for Controlled Image Captioning
- Lean Language Models Master Reasoning and Retrieval for Private AI Applications
Future Directions
While PMTFR marks a significant advancement, the researchers acknowledge areas for further exploration. The exact mechanisms by which injecting RAug-Rep activates latent reasoning abilities in the model are complex and warrant deeper investigation. This promising research direction could inspire future studies in the CIR community.
For more technical details, you can refer to the full research paper here.
