TLDR: This research introduces CoT Referring (CoTR), a new strategy that improves Multimodal Large Language Models (MLLMs) in understanding and localizing objects from complex language descriptions. By structuring training data to explicitly guide models through sequential reasoning steps, similar to human thought processes, CoTR helps MLLMs accurately identify objects in multi-step queries. The paper also presents RefLM, a new MLLM architecture, and a Composite Referring Benchmark for evaluating performance on challenging expressions, demonstrating significant accuracy improvements over existing methods.
Multimodal Large Language Models (MLLMs) have made significant strides in understanding both images and language, particularly in tasks like Referring Expression Comprehension (REC) and Referring Expression Segmentation (RES). These tasks involve identifying and localizing specific objects in an image based on a natural language description. While current models perform well with simple descriptions, their accuracy often declines sharply when faced with more complex, multi-step referring expressions.
A new research paper, titled “CoT Referring: IMPROVING REFERRING EXPRESSION TASKS WITH GROUNDED REASONING,” addresses this challenge by proposing a novel strategy called CoT Referring (CoTR). Authored by Qihua Dong, Luis Figueroa, Handong Zhao, Kushal Kafle, Jason Kuen, Zhihong Ding, Scott Cohen, and Yun Fu, this work introduces a structured, chain-of-thought training data approach to enhance how MLLMs reason across different modalities.
The Challenge of Complex Referring Expressions
Imagine a query like “the boy playing with a dog near the car.” For a human, this involves a sequence of steps: first locating the car, then identifying the dog near it, and finally finding the boy playing with that specific dog. Existing MLLMs often try to solve such queries in a single step, treating the description as a collection of objects rather than a structured sequence of dependencies. This single-shot approach leads to a deficit in compositional reasoning, causing performance drops in complex scenarios.
Introducing CoT Referring (CoTR)
Inspired by Chain-of-Thought (CoT) prompting in traditional Large Language Models, CoTR reformulates referring expressions to explicitly guide models through the necessary reasoning steps. Instead of just providing a final mask for the target object, CoTR data is structured as a chain of thought. This involves identifying relevant nouns (text anchors) and arranging them in a logical order that serves as anchor points for subsequent references. For instance, for the query “the boy playing with a dog near the car,” CoTR would guide the model to first locate the “car,” then the “dog,” and finally the “boy.”
A Novel Data Generation Pipeline
To create this structured CoTR data, the researchers developed an innovative four-stage pipeline that leverages a sequence of advanced language and vision-language models. This pipeline systematically decomposes textual structures into sequential referring steps, identifies relationships between objects, and ensures consistent reference alignment. It also includes rigorous validation steps at each stage to ensure high-quality annotations. This process generated approximately 20,000 high-quality CoTR instances for training.
Furthermore, recognizing the insufficiency of existing datasets for evaluating complex referring expressions, the team curated a new evaluation benchmark called the Composite Referring Benchmark. This benchmark specifically targets expressions containing three or more cross-related objects, providing a robust test for a model’s reasoning capabilities in challenging scenarios.
RefLM: A Unified MLLM Framework
To demonstrate the effectiveness of CoTR, the paper introduces RefLM, an MLLM that integrates detection and segmentation capabilities. Unlike some prior models, RefLM removes the need for a separate mask-decoder head. Instead, it predicts visual prompts (bounding boxes and points) that are then fed into the Segment Anything Model (SAM) to generate precise masks. This approach simplifies the architecture and allows for direct supervision at the token level.
RefLM also incorporates a novel adaptive weighted loss function during training. This loss prioritizes the final target object’s accuracy while still supervising the intermediate anchor objects, ensuring that the model focuses on the primary objective without neglecting the reasoning steps.
Promising Results
Experimental results on both the newly curated Composite Referring Benchmark and existing datasets like RefCOCO/+/g show the significant effectiveness of the CoTR approach. RefLM models consistently outperform baseline models, with a notable increase in accuracy, especially on complex referring expressions. The ablation studies further confirm that the improvements are due to the reasoning knowledge and structure embedded by CoTR, not just additional supervision.
Also Read:
- Unlocking Human-Object Interaction Detection with Language Models
- Making AI Reasoning Clearer in Visual Question Answering
Conclusion
The introduction of CoT Referring and the RefLM model represents a significant step forward in improving how MLLMs handle complex visual grounding tasks. By explicitly modeling sequential reasoning, this work not only enhances localization accuracy but also provides a new benchmark to encourage further research into the reasoning capabilities of MLLMs. You can read the full paper here: CoT Referring: IMPROVING REFERRING EXPRESSION TASKS WITH GROUNDED REASONING.
