TLDR: The Chain of Questions (CoQ) framework enables multimodal language models to proactively generate questions about their surroundings, guiding them to selectively activate relevant sensory modalities (vision, audio, spatial) to gather necessary information. This approach significantly enhances reasoning, interpretability, and accuracy in complex real-world scenarios. Evaluated on a novel benchmark dataset, the CoQ method improves a foundation model’s ability to integrate pertinent sensory information, marking a step towards more contextually aware AI.
Large Language Models (LLMs) have made incredible strides in understanding and generating human language, especially with techniques like Chain-of-Thought, which help them break down complex problems into step-by-step reasoning. These advancements have significantly improved how accurate and understandable LLM outputs are, particularly for text-based tasks.
However, a major challenge remains: these powerful models are often limited to just text. They don’t naturally interact with the rich, diverse information from the real world, which includes sights, sounds, and spatial awareness. Humans, on the other hand, constantly integrate multiple senses to make sense of their surroundings – think about navigating a busy street, where you’re simultaneously processing visual cues, auditory information, and your spatial position.
Current multimodal language models (MLLMs) typically treat non-textual information as secondary inputs, passively incorporating them. This passive approach limits their ability to actively decide what additional sensory information they need for a given task, making them less effective in dynamic, real-world situations.
Introducing the Chain of Questions (CoQ) Framework
To address these limitations, researchers Nima Iji and Kia Dashtipour from Edinburgh Napier University have introduced a novel approach called the Chain of Questions (CoQ) framework. This framework is designed to encourage multimodal language models to proactively generate curiosity-driven questions about their environment. These questions then guide the model to selectively activate relevant sensory modalities, such as vision, audio, or spatial perception, to gather the critical information needed for accurate reasoning and response generation.
The CoQ framework operates through a structured pipeline, moving from a user’s initial prompt to activating specific sensors:
- Prompt: The initial text input from the user.
- Question: The model generates curiosity-driven questions to gather more multimodal data (e.g., “What do I see?”, “What am I hearing?”).
- Task: Each question triggers a specific operation, like object detection, speech-to-text, or spatial detection.
- Sensor: These tasks activate the necessary hardware or software-based modalities, such as cameras, microphones, or LiDAR sensors.
Once all the observations are collected through these sensors, they are combined to form a comprehensive multimodal context. This enriched context then allows the model to generate a more structured and grounded response, mirroring how humans inquire and perceive their environment.
A New Benchmark for Multimodal Curiosity
To evaluate the CoQ framework, the researchers developed a unique multimodal benchmark dataset. They integrated several existing datasets, including WebGPT (for textual prompts), ScienceQA (for prompts with and without visual evidence), AVSD (for audio-visual dialogues in video sequences), and ScanQA (for spatial information from 3D indoor scans). This comprehensive dataset, totaling over 180,000 instances, allows for rigorous testing of a model’s ability to identify when and what kind of additional multimodal information is necessary.
Also Read:
- Object-Aware Reasoning: A New Approach to Audio-Visual Segmentation
- Unraveling Why AI Reasoning Models Struggle with Complex Multi-Hop Questions
Experimental Insights
Experiments were conducted using various language models, including FLAN T5 models of different sizes (base, large, XL) and Llama 2. The primary goal was to assess how accurately and relevantly these models generated curiosity-driven questions. The results showed that the FLAN T5 XL model (3 billion parameters) achieved the highest accuracy in producing relevant multimodal questions aligned with the input prompts. Smaller models like FLAN T5 base and large were less effective in generating targeted questions, though FLAN T5 models generally displayed higher overall curiosity compared to Llama 2.
These findings highlight that both the model’s architecture and its size significantly influence the successful implementation of the CoQ framework. The Chain of Questions framework represents a significant step towards creating more sophisticated, contextually aware language models capable of effectively operating in complex real-world environments. For more details, you can read the full research paper here.
