TLDR: A study fine-tuned a Vision-Language Model (VLM), Qwen2.5-VL-7B, to accurately convert complex financial tables from Malaysian audited reports into Markdown format. The model achieved high accuracy (92.20%) and structural fidelity (96.53% Markdown TEDS) on a curated dataset, significantly outperforming larger and proprietary models like GPT-4o and Gemini 2.5 Flash, while also reducing inference time. This demonstrates the effectiveness of domain-specific fine-tuning for specialized document understanding tasks.
Extracting accurate information from financial documents, especially complex tables, has long been a significant challenge in the field of document understanding. This is particularly true for regulatory and analytical purposes, where even minor errors can have substantial implications. A recent study tackles this very issue, focusing on the intricate task of converting financial tables from Malaysian audited financial reports into a structured Markdown format.
The complexity arises from several factors unique to these reports, including tables with rotated layouts, multi-level headers, and subtle, implicit structural cues that are easily missed by conventional systems. To overcome these hurdles, researchers proposed a novel approach: fine-tuning a Vision-Language Model (VLM) specifically for this task. The chosen base model was Qwen2.5-VL-7B, which was then optimized for generating high-fidelity Markdown directly from document images.
The Markdownification Pipeline
The core of their approach involves a process they call “Markdownification.” This pipeline converts diverse financial tables into a standardized Markdown representation. This involves flattening hierarchical headers, preserving distinctions between multiple entities (like ‘Group’ and ‘Company’) and time periods (‘2023’, ‘2022’), and explicitly identifying implicit structural elements such as note indicators. By standardizing these varied layouts into a consistent text format, the reliability of subsequent financial analysis performed by other language models is significantly enhanced.
To train their model, the team meticulously curated a dataset of 2,152 image-text pairs. This dataset included augmentations, such as rotating 30% of the existing entries by 90° or 270° to help the model handle common rotated table layouts found in financial reports. The fine-tuning itself was performed using a supervised strategy called LoRA (Low-Rank Adaptation), which is an efficient method for adapting large language models to specific tasks with limited computational resources.
Evaluating Performance
Assessing the model’s performance required a robust evaluation framework. The researchers employed a dual approach: a criteria-based LLM-as-a-judge system and a new metric called Markdown Tree-Edit-Distance-based Similarity (TEDS). The LLM-as-a-judge, powered by OpenAI’s o3-mini, evaluated the generated Markdown against five key criteria: correct row count, correct column count, semantically accurate headers, correct item order, and valid Markdown formatting. This provided a fine-grained assessment of accuracy.
The Markdown TEDS metric offered a holistic score for structural fidelity. It works by representing tables as tree structures and calculating the edit distance between the model’s output and the ground truth. This metric is particularly effective because it accounts for both structural and content errors, providing a single, comprehensive score for how well the model reconstructs the table. The researchers also introduced novel modifications to TEDS, including table structure isolation, fuzzy table merging (to handle cases where models fragment a single logical table), and optimal multi-table matching, ensuring accurate evaluation even with complex outputs.
Also Read:
- Fourier-VLM: A New Approach to Efficient Vision-Language Models
- New Benchmark Reveals AI’s Struggle with Causal Reasoning in Infographics
Impressive Results
The results were compelling. The fine-tuned Qwen2.5-VL-7B model achieved an impressive 92.20% overall accuracy on the criteria-based assessment and a 96.53% Markdown TEDS score. This performance significantly surpassed its base model, which only managed 32.80% accuracy and 52.08% TEDS. More notably, the specialized model also outperformed larger-scale Vision-Language Models, such as Qwen2.5-VL-32B and Qwen2.5-VL-72B AWQ, as well as specialized reasoning-enabled models.
Perhaps the most striking finding was its superiority over widely used proprietary models like OpenAI’s GPT-4o and Gemini 2.5 Flash. The fine-tuned model achieved higher accuracy than both, demonstrating that a lightweight, specialized open-source model can deliver state-of-the-art results for this domain-specific task. Furthermore, it significantly reduced inference time compared to other self-hosted alternatives, making it a more efficient solution.
This study underscores the immense value of domain-specific fine-tuning for specialized document understanding tasks. By strategically adapting an open-source VLM to the unique challenges of financial tables, the researchers have created a high-performance, cost-effective, and transparent alternative to larger, more generalist, and often closed-source models. This advancement helps bridge the gap between unstructured financial documents and downstream automation, paving the way for more reliable and scalable financial analysis. You can read the full research paper here.
