TLDR: EaGERS is a novel, training-free, and model-agnostic pipeline for Document Visual Question Answering (DocVQA). It improves the transparency and reproducibility of Vision-Language Models by generating natural language explanations, identifying relevant document sub-regions through multimodal embedding similarities and majority voting, and then re-querying the model on a masked image to ensure answers are derived solely from these validated regions. Experiments show it outperforms base models in accuracy while providing clear answer grounding.
In the rapidly evolving field of Document Visual Question Answering (DocVQA), where artificial intelligence models answer questions based on document images, a new approach called EaGERS has emerged. Developed by Maximiliano Hormazábal, Héctor Cerezo-Costas, and Dimosthenis Karatzas, EaGERS aims to make these powerful Vision-Language Models (VLMs) more transparent and reliable without requiring extensive fine-tuning or complex prompt engineering.
Current general-purpose VLMs, while capable of understanding documents, often struggle with providing clear justifications for their answers. This lack of “grounding” – knowing exactly which part of the document led to an answer – can be a significant hurdle for their deployment in real-world applications. EaGERS addresses this by introducing a fully training-free and model-agnostic pipeline that ensures answers are derived solely from explicitly identified and relevant regions of a document.
How EaGERS Works: A Three-Stage Pipeline
The EaGERS pipeline operates in three distinct stages, designed to enhance explainability and reproducibility:
First, the system performs Explanation Generation. A VLM is given the document image and a question. Instead of directly answering, it generates a natural language explanation describing how to find the answer visually within the document. This explanation acts as a semantic guide for the subsequent steps.
Next is Region Selection. The document image is divided into a grid (for example, 5×5 or 5×10 cells). Each of these sub-regions is then converted into a vector representation. To ensure robustness and mitigate biases, EaGERS uses an ensemble of three multimodal models – BLIP, CLIP, and ALIGN – to create these embeddings. The system then compares the embeddings of the generated explanation with those of each sub-region using cosine similarity. Through a majority voting process across the different embedders, the most relevant sub-regions are identified and selected. This step precisely pinpoints the areas of the document that are crucial for answering the question.
Finally, Masking and Re-query takes place. A new version of the original image is created where all parts outside the selected relevant regions are blacked out, effectively masking them. This “masked image,” along with the original question, is then fed back into the same VLM. The model is forced to generate its answer using only the information visible within the justified regions, ensuring that the final answer is directly traceable to specific parts of the document. This process significantly enhances transparency and reproducibility.
Performance and Impact
Experiments conducted on the DocVQA dataset demonstrated that EaGERS not only matches but often outperforms the base VLM in terms of Exact Match (EM) accuracy and Average Normalized Levenshtein Similarity (ANLS). Notably, configurations using a finer grid (50 cells) combined with a slight margin expansion (15%) around the selected regions yielded the best results, suggesting that precise spatial granularity and a small overlap can significantly improve the localization and understanding of relevant information. While the pipeline does introduce a modest increase in inference time compared to the base model, this is a trade-off for the substantial gains in explainability and answer grounding.
The researchers acknowledge that while the fixed grid approach simplifies reproducibility for this initial study, future work will explore more flexible, adaptive grids to better handle documents with irregular layouts. They also plan to investigate more robust fusion strategies for region selection and evaluate the system against even more advanced state-of-the-art models. The ultimate goal is to further refine the system to ensure even greater accuracy and reliability in its spatial explanations.
Also Read:
- Unlocking Transparency in Table Question Answering with Code
- Enhancing Video Understanding: A New Approach to Fundamental Question Generation
A Step Towards More Trustworthy AI
EaGERS represents a significant step forward in making AI models for document understanding more transparent and trustworthy. By explicitly linking answers to specific visual regions and providing natural language explanations, it offers a clear audit trail for how an answer was derived. This capability is crucial for enterprise applications where accountability and interpretability are paramount. For more details, you can refer to the full research paper here.
