TLDR: CAD2DMD-SET is a new tool that generates synthetic datasets of digital measurement devices (DMDs) to fine-tune Large Vision-Language Models (LVLMs). It uses 3D CAD models, advanced rendering, and image composition to create diverse, VQA-labelled data. Alongside, DMDBench, a real-world validation set, was created. Fine-tuning LVLMs with CAD2DMD-SET data significantly improved their performance in reading DMDs under challenging conditions like clutter, occlusions, and motion blur, with one model showing a 200% score increase. The tool is set to be open-source.
Large Vision-Language Models (LVLMs) have shown remarkable progress in various multimodal tasks, but they often struggle with seemingly simple scenarios like accurately reading values from Digital Measurement Devices (DMDs). This difficulty is particularly evident in real-world conditions, which often include visual clutter, obstructions, extreme viewing angles, and motion blur, common issues in head-mounted cameras and Augmented Reality (AR) applications.
Addressing these limitations, a new tool called CAD2DMD-SET has been introduced. This innovative synthetic data generation tool is specifically designed to enhance LVLMs’ performance in visual question answering (VQA) tasks involving DMDs. By leveraging 3D CAD models, advanced rendering techniques, and high-fidelity image composition, CAD2DMD-SET produces diverse, VQA-labelled synthetic DMD datasets ideal for fine-tuning LVLMs.
In addition to the synthetic data generation tool, the researchers also developed DMDBench, a carefully curated validation set comprising 1,000 annotated real-world images. This dataset is crucial for evaluating how well models perform under practical, challenging conditions. The study benchmarked three state-of-the-art LVLMs using the Average Normalised Levenshtein Similarity (ANLS) metric. After fine-tuning these models with datasets generated by CAD2DMD-SET, significant improvements were observed. For instance, InternVL showed a remarkable score increase of 200% without negatively impacting its performance on other tasks. This compelling result demonstrates that the CAD2DMD-SET training dataset substantially boosts the robustness and accuracy of LVLMs when faced with the challenging conditions mentioned earlier.
The CAD2DMD-SET Pipeline
The CAD2DMD-SET tool operates through a sophisticated pipeline consisting of three main modules:
Display Generator: This module creates various display images for each device. It automatically generates dictionaries containing the possible measurement ranges for specific DMDs and then uses these ranges to produce display images with varying measurements. It processes real display images to isolate the foreground, embeds new measurement values into predefined regions of interest (ROIs) using a selection of segmented-style fonts, and saves the synthetic display images.
Renderer: This component programmatically stitches the generated display images onto 3D CAD models of the devices within Blender. It introduces variability in lighting conditions, model orientation, and camera distance to enhance visual diversity. The module performs UV mapping to project synthetic display images onto the device surfaces and generates corresponding VQA labels and segmentation masks. Post-processing steps include applying motion blur to simulate dynamic capture conditions and generating binary segmentation masks from depth maps.
Image Composer: The final module uses the libcom library to create high-fidelity composite images. It places the rendered CAD models into realistic indoor background scenes. The FOPA model within libcom helps determine the most contextually appropriate placement and scale for the foreground objects, ensuring a realistic integration. While image harmonization was explored, it was found to reduce the clarity of digital displays, so it remains an optional feature. The Image Composer also supports multiprocessing to accelerate data generation.
Experiments and Results
For the synthetic training dataset, six diverse DMDs were selected, including a power supply, a multimeter, a metronome, a thermometer, a pulse oximeter, and a blood pressure monitor. Their 3D CAD models were reconstructed using photogrammetry. The CAD2DMD-SET tool then generated a training dataset of 100,000 composite images with VQA labels for fine-tuning LVLMs.
DMDBench, the validation dataset, consists of 1,000 manually annotated real images of selected DMDs, designed to emulate realistic and imperfect conditions such as variations in scale, orientation, lighting, background clutter, and motion blur. The models benchmarked were Pixtral-12B, LLaVA-1.5-13B, and InternVL2.5-26B. Both LLaVA and InternVL were evaluated before and after fine-tuning using LoRA modules, with a mixture of standard instruction tuning data and the synthetic dataset.
The results clearly showed that DMDBench is a challenging benchmark for non-fine-tuned LVLMs. However, the addition of synthetic data from CAD2DMD-SET during the instruction tuning phase drastically improved the ANLS scores. For example, InternVL’s ANLS score increased from 32.92% to 96.04% when trained with 50% of its data mixture. These improvements were attributed to better identification of DMDs and more accurate measurement readings under difficult environmental conditions. The fine-tuning also maintained stable performance on external datasets like TextVQA, indicating that catastrophic forgetting was avoided.
Also Read:
- MMSearch-Plus: A New Benchmark Elevates the Challenge for AI Web Browsing
- PosterForest: A New Framework for Automated Scientific Poster Creation
Ablation Studies
Several ablation studies were conducted to understand the impact of specific CAD2DMD-SET features:
- Object Placement and Scale: Fine-tuning with FOPA-guided realistic object placement yielded significantly better ANLS scores (95.16%) compared to random placement (78.17%), despite the latter being faster to generate.
- Motion Blur: Including motion blur in 20% of the training images resulted in higher ANLS scores (95.16%) than training without any motion blur (88.89%), highlighting its importance for robustness in real-world conditions.
- One-word Label Format: Training with a one-word label format (measurement or unit only) improved word-level accuracy by approximately 8%, primarily due to a 15% gain in unit-level accuracy, suggesting this format is more beneficial for OCR capabilities.
- Inter-device Generalization: Even when a specific device (power supply) was excluded from the training set, the fine-tuned model still showed substantial improvement (from 41.58% to 72.94% ANLS) on images of that device, demonstrating the tool’s ability to foster inter-device generalization.
In conclusion, CAD2DMD-SET offers an automated pipeline for generating synthetic datasets that significantly improve LVLMs’ ability to read digital measurement devices in complex visual conditions. The accompanying DMDBench validation set provides a robust benchmark for assessing model performance under realistic challenges. The tool is expected to be released as open-source, allowing the community to expand its capabilities. For more details, you can read the full research paper here.
