TLDR: A new framework utilizes multiple AI agents to automatically generate high-quality, contextually relevant question-answer pairs from extensive and complex documents in both English and Arabic. This innovation addresses the critical shortage of fine-grained training data for Large Vision-Language Models (LVLMs), particularly for low-resource languages, thereby significantly enhancing their ability to comprehend long-context information and improving scalability in data annotation.
Understanding long and complex documents, especially those with intricate layouts, has been a significant hurdle in the field of vision-language research. While advanced Large Vision-Language Models (LVLMs) perform well on shorter texts, their accuracy drops considerably when dealing with extensive documents. A major reason for this decline is the lack of detailed training data, particularly for languages with fewer digital resources, such as Arabic. Traditional methods of creating this data rely heavily on human annotation, which is both expensive and time-consuming.
To address these challenges, researchers have introduced a new, fully automated framework that uses multiple AI agents to generate high-quality questions for long documents. This innovative approach efficiently creates both single-page and multi-page questions for hundreds of pages of documents in both English and Arabic, covering a wide range of subjects. This development is crucial for enhancing the ability of LVLMs to understand long-context information.
The core of this framework is its multi-agent interactive system. It begins with a thorough preprocessing of PDF documents, converting each page into an image, applying advanced OCR (Optical Character Recognition) using LVLMs, and analyzing the document’s layout to identify structural elements like headings, paragraphs, and figures. The documents are then divided into manageable sections, ensuring context is maintained across pages.
Also Read:
- Connecting Images and Text for Smarter AI: Introducing MMGraphRAG
- Beyond Relevance: How AI Models Are Learning to Pick Truly Useful Information for Better Answers
How the Multi-Agent System Works
Once the documents are preprocessed, a chain of specialized AI agents works together in an iterative loop to generate and refine questions and answers:
-
Question Generation Agent: This agent creates initial questions based on the document’s content and layout.
-
Question Extraction Agent: It filters these generated questions, removing any that are irrelevant or repetitive.
-
Answer Generation Agent: This agent produces candidate answers directly from the document content for the filtered questions.
-
Assessment and Feedback Agent: This crucial agent evaluates the quality of the generated questions and answers, estimates their difficulty, and provides feedback to guide further improvements. If the accuracy of answers is high, it prompts the question generation agent to create more complex questions.
-
Evidence Validation Agent: As a final step, this agent ensures that the generated question-answer pairs are consistent and align with the supporting evidence in the document, preventing mismatches seen in other datasets.
This automated, end-to-end process significantly boosts scalability and reduces the time needed for data annotation, especially for low-resource languages like Arabic, where data scarcity has been a major barrier for LVLMs.
The framework has been used to create a new dataset called AraEngLongBench, which includes a diverse set of question types, with reasoning questions being the most common, followed by hypothetical reasoning. The answers generated also vary widely, from text and integers to code and floats, demonstrating the dataset’s ability to test different cognitive abilities.
Experiments conducted on the AraEngLongBench dataset show that closed-source models like Gemini-1.5 Pro and Gemini-2.0 Flash currently lead in performance, particularly for English tasks. GPT-4o also performs well, showing a more cautious approach to unanswerable questions. Among open-source models, Qwen 2.5 VL is rapidly improving, especially for English tasks. However, a consistent finding across all tested models is their struggle with Arabic understanding tasks and their difficulty in confidently identifying unanswerable questions. This highlights the need for future work in cross-lingual generalization and improving confidence calibration in LVLMs.
This research marks a significant step forward for the Document Understanding community, particularly for long-context scenarios. By providing a robust and diverse training dataset, it lays a strong foundation for developing more capable LVLMs for real-world applications. For more technical details, you can refer to the full research paper: Multi-Agent Interactive Question Generation Framework for Long Document Understanding.
