TLDR: MACT is a multi-agent framework for visual document understanding and question answering. It uses four specialized agents (planning, execution, judgment, answer) and innovative strategies like a dedicated judgment agent for self-correction, mixed reward modeling, and agent-wise test-time scaling. MACT achieves superior performance on document-based tasks, especially those with long visual contexts and complex reasoning, outperforming larger models with a smaller parameter scale.
In the rapidly evolving field of artificial intelligence, Vision-Language Models (VLMs) are at the forefront of enabling machines to understand and interact with visual information, particularly documents. However, existing VLMs, whether designed for general tasks or specialized document understanding, often face significant hurdles. These challenges include limitations imposed by their parameter size, a lack of robust self-correction abilities, and suboptimal performance when dealing with lengthy visual contexts and intricate reasoning tasks, which are common in document-based applications.
To tackle these issues, researchers have introduced a novel framework called MACT, which stands for Multi-Agent Collaboration framework with Test-Time scaling. MACT is specifically designed to enhance visual document understanding and visual question answering (VQA). What makes MACT unique is its architecture, which comprises four distinct, small-scale agents: a planning agent, an execution agent, a judgment agent, and an answer agent. Each agent is assigned a clear role, and they work together seamlessly to process information.
How MACT Works: The Collaborative Agents
The MACT framework operates through the synergistic collaboration of its four agents:
-
Planning Agent: This agent is responsible for analyzing the original question and breaking it down into high-level execution plans. It can even generate multiple relevant sample plans to explore diverse pathways to a solution, ensuring a comprehensive approach without getting bogged down in implementation details.
-
Execution Agent: Following the plans laid out by the planning agent, the execution agent carries out each step. It utilizes a tool library to perform tasks and generates a detailed execution process, passing the results to the next stage.
-
Judgment Agent: A standout feature of MACT is its dedicated judgment agent. Unlike conventional systems where correction is often integrated with generation, this agent solely focuses on verifying the correctness of the execution plans and processes. If it detects any mistakes, it identifies the problematic step and redirects the issue back to the relevant preceding agent (planning or execution) for revision. This separation of judgment from correction introduces a neutral, unbiased review process, leading to more accurate and efficient self-correction.
-
Answer Agent: Once the judgment agent confirms the correctness of the process, the answer agent synthesizes all the information, including any corrected segments, to formulate the final, accurate answer to the question.
Key Innovations for Enhanced Performance
Beyond its multi-agent structure, MACT incorporates two crucial innovations:
-
Mixed Reward Modeling: To optimize the learning process for each agent and the framework as a whole, MACT uses a mixed reward strategy. This approach combines agent-specific rewards, which provide immediate feedback for individual tasks, with a global outcome reward. The global reward encourages overall collaboration and guides the model towards correct solutions, preventing agents from acting selfishly.
-
Agent-Wise Hybrid Test-Time Scaling: Recognizing that different agents have distinct functions, MACT employs a customized test-time scaling strategy for each. For instance, the planning agent generates multiple parallel plans, increasing the chances of finding a correct path. The execution agent produces several candidate executions for each step, selecting the best one. The judgment agent uses a “budget forcing” method to ensure thorough logical analysis. This tailored scaling significantly boosts the framework’s ability to handle long visual contexts and complex reasoning tasks.
Also Read:
- A New Approach to Radiology Question Answering Using AI Agents
- Enhancing Visual Reasoning Reliability in AI Models with ViFP
Impressive Results and Future Outlook
Evaluated across 15 diverse benchmarks, including both document-based and non-document-based settings, MACT has demonstrated superior performance. Despite having a smaller parameter scale (under 30 billion parameters) compared to many larger models, MACT variants consistently outperform state-of-the-art open-source and even closed-source models. Notably, MACT models secured the top three positions in average scores and led in 13 out of 15 benchmarks. They showed particular strength in tasks involving long visual contexts and complicated reasoning, areas where many existing VLMs struggle.
The MACT framework represents a significant step forward in visual document understanding and question answering. By leveraging a collaborative multi-agent system, innovative self-correction mechanisms, and tailored test-time scaling, it unlocks the full potential of smaller-parameter models, proving that efficiency can go hand-in-hand with superior performance. The code for MACT will be made available, fostering further research and development in this promising direction. For more details, you can refer to the research paper: Visual Document Understanding and Question Answering: A Multi-Agent Collaboration Framework with Test-Time Scaling.
