TLDR: A new three-stage hybrid AI framework automates the interpretation of complex 2D multi-view engineering drawings. It uses YOLOv11 models for layout segmentation (views, title blocks, notes) and fine-grained annotation detection (measures, GD&T, surface roughness). Two specialized Donut-based Vision Language Models then semantically parse textual and numerical content. This OCR-free approach converts drawing information into a structured JSON format, significantly improving accuracy and efficiency for integration with CAD and manufacturing systems.
Engineering drawings are the backbone of manufacturing, serving as the universal language for conveying design intent, tolerances, and production specifics. However, the manual interpretation of complex multi-view drawings, especially those with dense annotations, has long been a bottleneck in digital manufacturing workflows. Traditional methods, including generic optical character recognition (OCR) systems, often struggle with varied layouts, rotated text, and the intricate mix of symbols and text found in these documents.
To address these significant challenges, a new multi-stage hybrid framework has been developed for the automated interpretation of 2D multi-view engineering drawings. This innovative approach leverages modern detection and vision language models (VLMs) to convert complex visual information into structured, machine-readable formats.
A Three-Stage Approach to Understanding Drawings
The proposed framework operates through three distinct stages, each designed to tackle a specific aspect of drawing interpretation:
Stage 1: Layout Segmentation
The initial stage focuses on understanding the overall structure of the drawing. It uses a detection model, YOLOv11-det, to segment the drawing into key regions such as individual views, the title block, and notes sections. This step is crucial as it provides spatial context and simplifies subsequent processing by isolating different functional areas of the drawing.
Stage 2: Fine-Grained Annotation Localization
Once the main regions are identified, the second stage delves into the details within each detected view. Here, an orientation-aware object detection model, YOLOv11-obb, is employed. This model is specifically designed to detect fine-grained annotations like measures (dimensions, radii), Geometric Dimensioning and Tolerancing (GD&T) symbols, and surface roughness indicators, even when they are rotated or oriented at various angles. Processing one view at a time helps prevent confusion from overlapping annotations.
Stage 3: Vision Language Parsing
The final stage is where the semantic content of the localized regions and annotations is interpreted. This stage utilizes two specialized Donut-based Vision Language Models (VLMs). An ‘Alphabetical VLM’ is used to extract textual and categorical information from title blocks and notes, handling free-form text. A ‘Numerical VLM’ is dedicated to interpreting quantitative data, such as the values associated with measures, GD&T frames, and surface roughness symbols. This VLM was fine-tuned to accurately extract numerical and symbolic specifications.
Robust Performance and Data Integration
To ensure the framework’s robustness and generalization, two specialized datasets were developed: 1,000 drawings for layout detection and 1,406 for annotation-level training. The models demonstrated strong performance, with the layout detection achieving high accuracy for views, title blocks, and notes. The annotation localization also performed well for measures and GD&T symbols, though surface roughness detection showed slightly lower accuracy due to dataset imbalance.
The Numerical VLM achieved an impressive F1 score of 0.963, proving highly effective for quantitative interpretation. While the Alphabetical VLM’s overall F1 score was 0.672, indicating that text-heavy and categorical fields remain a challenge for OCR-free models, it still provided valuable insights.
A key outcome of this framework is the generation of a unified JSON output. This structured data can be seamlessly integrated into Computer-Aided Design (CAD), process planning, and manufacturing databases, effectively bridging the gap between traditional engineering drawings and modern digital manufacturing ecosystems. This eliminates the need for manual transcription, making the process scalable and efficient.
Also Read:
- From Sketch to Structure: Automating 3D BIM Creation with Human-AI Collaboration
- AI Framework Accelerates Post-Earthquake Structural Damage Assessment
Looking Ahead
While the framework represents a significant leap forward, the researchers acknowledge areas for future improvement. Enhancing the performance of textual data extraction, particularly in complex title block fields, and addressing the dataset imbalance for less common annotations like surface roughness are key priorities. Future work includes integrating advanced VLMs like GPT-4o for better text comprehension and exploring synthetic data augmentation to bolster training for underrepresented annotations.
This multi-stage hybrid framework offers a reliable and efficient alternative to generic OCR-driven solutions, paving the way for intelligent, self-improving systems that can accelerate digital transformation across manufacturing industries. You can read the full research paper here.
