TLDR: This research introduces a new method to improve Question-Answering (QA) from technical telecom documents, especially when answers are hidden in flowcharts. Traditional text-based QA systems often fail with images. The proposed approach converts flowcharts into graph structures using fine-tuned Visual Language Models (VLMs). These graph representations are then integrated into a text-based Retrieval Augmented Generation (RAG) system. The study shows that this method significantly improves information retrieval from flowcharts, making QA systems more robust and cost-effective by reducing the need for VLMs during inference.
Technical documents, especially in specialized fields like telecommunications, often contain crucial information embedded not just in text, but also within figures such as flowcharts and diagrams. While modern Question-Answering (QA) systems, particularly those based on Retrieval Augmented Generation (RAG), have become quite advanced for text-based queries, they frequently struggle when the answer lies within an image.
A recent research paper, titled “A Graph-based Approach for Multi-Modal Question Answering from Flowcharts in Telecom Documents,” addresses this challenge directly. Authored by Sumit Soman, H. G. Ranjani, Sujoy Roychowdhury, Venkata Dharma Surya Narayana Sastry, Akshat Jain, Pranav Gangrade, and Ayaaz Khan, the paper introduces an innovative method to extract and utilize information from flowcharts for more effective QA in the telecom domain. You can read the full paper here: Research Paper.
The Problem with Images in QA
Traditional RAG systems primarily focus on text. When a question’s answer is visually represented, such as in a flowchart detailing a process or decision tree, these systems often fall short. Visual Language Models (VLMs) exist for visual QA, but they are typically trained on general datasets (like images of everyday objects) and are not well-suited for the highly specific and text-heavy diagrams found in technical telecom documents. Furthermore, using large VLMs for every query in a deployed system can be very expensive due to their computational demands.
A Graph-Based Solution
The researchers propose a clever solution: converting flowcharts into structured graph representations. Imagine a flowchart as a series of steps or decisions. In this approach, each step or decision box becomes a ‘node’ in a graph, and the connections between them become ‘edges’. Any text within the boxes or along the connecting lines is captured as attributes of these nodes and edges.
The end-to-end process involves several key steps:
- Image Classification: First, the system automatically identifies and categorizes images within technical documents. This helps in pinpointing flowcharts among other types of diagrams like block diagrams or screenshots. A fine-tuned Document Image Transformer (DIT) model is used for this classification.
- Flowchart to Graph Conversion: Once a flowchart is identified, a specially fine-tuned Visual Language Model (VLM) is employed to convert the image into a graph structure, typically represented in a JSON format. This VLM is trained on synthetic flowcharts to ensure it understands the specific elements and relationships within these diagrams, even for telecom-specific content. The fine-tuning significantly improves the accuracy of the generated graph, as measured by a metric called Graph Edit Distance (GED).
- Integrating Graphs into RAG: The generated graph structures, which are essentially text-based (JSON), are then processed and embedded into a vector database. This allows them to be retrieved by standard text-based embedding models, just like regular text. The paper explores different ways to ‘chunk’ these graph structures (e.g., embedding each node separately, all nodes together, or the entire graph JSON) to find the most effective method for retrieval.
Key Advantages and Findings
One of the significant benefits of this approach is that it reduces the reliance on large, expensive VLMs during the inference (question-answering) phase. Once the flowchart is converted to a graph, a standard text-based RAG system can handle the retrieval, leading to cost savings for deployed QA systems.
The study benchmarked its approach on a proprietary telecom QA dataset. Results showed that the fine-tuned VLM produced graph representations with significantly lower Graph Edit Distance (meaning they were very close to the true structure of the flowcharts). For retrieval performance, the domain-adapted embedding model, TeleRoBERTa, generally performed better than a publicly available model. Interestingly, embedding the entire graph’s JSON as a single chunk often yielded higher retrieval accuracy, especially for questions related to specific nodes within the flowchart.
Also Read:
- Unlocking Smarter AI Responses: The SemRAG Approach to Knowledge Integration
- Bridging the Gap: How AI Helps Access Manufacturing Knowledge Graphs
Looking Ahead
This research marks a significant step towards making QA systems more capable of handling the diverse information formats found in complex technical documents. Future work will likely explore how these graph structures can be used more effectively by the text generator component of RAG systems and extend the approach to other types of structured diagrams like UML Sequence diagrams.
