TLDR: ToolVQA is a new large-scale dataset (23K samples) for Visual Question Answering (VQA) that helps AI models better use external tools for multi-step reasoning in real-world scenarios. It uses a novel data generation pipeline called ToolEngine to create complex, implicit multi-step tasks across 10 diverse tools. Fine-tuning models like LLaVA-7B on ToolVQA significantly improves their performance and generalization, even outperforming GPT-3.5-turbo on several unseen tasks.
The integration of external tools into Large Foundation Models (LFMs) is a promising avenue for enhancing their problem-solving abilities. While existing studies have shown strong performance in tool-augmented Visual Question Answering (VQA), there remains a significant gap in real-world tool-use proficiency, especially in diverse multimodal settings that demand multi-step reasoning.
To address this, researchers Shaofeng Yin, Ting Lei, and Yang Liu have introduced ToolVQA, a large-scale multimodal dataset comprising 23,000 samples. Unlike previous datasets that often rely on synthetic scenarios and simplified queries, ToolVQA features real-world visual contexts and challenging implicit multi-step reasoning tasks, making it much more aligned with how real users interact with AI systems.
Introducing ToolEngine: A Novel Data Generation Pipeline
To construct ToolVQA, the researchers developed ToolEngine, a novel data generation pipeline. This engine employs an image-guided Depth-First Search (DFS) approach combined with a Longest Common Subsequence (LCS)-based example matching mechanism. This sophisticated method simulates human-like tool-use reasoning, allowing for the generation of complex, multi-step reasoning tasks. ToolEngine ensures that the generated data reflects real user needs, uses real-world tool outputs (which can be noisy), and involves logically connected multi-step reasoning processes.
ToolVQA encompasses 10 multimodal tools across 7 diverse domains. These tools cover a wide range of functions, including perception (like ImageCaption and ObjectDetection), operation (like GoogleSearch), logic (such as Calculator), and creativity (like TextToImage). On average, each sample in the dataset requires 2.78 reasoning steps, highlighting its inherent complexity.
Also Read:
- Refine-IQA: A Multi-Stage Approach to Enhancing Image Quality Assessment in AI Models
- Chain of Questions: Empowering Language Models with Multimodal Curiosity
Performance and Generalizability
The quality of ToolVQA was rigorously assessed, with a human validation accuracy of 90.8% on a randomly sampled subset of the training data. The test set, consisting of 2,550 samples, was carefully filtered and fully human-annotated.
Experiments demonstrated that fine-tuning the LLaVA-7B model on ToolVQA significantly enhances its performance. This fine-tuned model not only achieved impressive results on the ToolVQA test set but also surpassed the performance of the larger, closed-source GPT-3.5-turbo model on five out-of-distribution (OOD) datasets. These OOD benchmarks included tasks and tools that the model had not encountered during its training, showcasing the strong generalizability of the ToolVQA-trained agent in real-world tool-use scenarios.
The research also shed light on key challenges for LFMs, particularly their struggles with integrating new information introduced in multi-turn dialogues and accurately predicting arguments for tools. While fine-tuning significantly improved instruction formatting and tool selection, argument prediction and answer summarization remain areas for further improvement, as they require a deeper understanding and generalization capability from the models.
This work represents a significant step forward in developing more capable and generalizable AI agents that can effectively use tools to solve complex, real-world visual questions. For more details, you can refer to the full research paper: ToolVQA: A Dataset for Multi-step Reasoning VQA with External Tools.
