TLDR: Sketch2BIM is a new pipeline that converts hand-drawn floor plans into 3D BIM models using multimodal large language models (MLLMs) and a multi-agent system. It incorporates human feedback to iteratively refine initial extractions of walls, doors, and windows from sketches into a structured JSON, which is then translated into an executable script for Autodesk Revit. The system effectively handles complex, non-Manhattan layouts, achieving high accuracy in detection and geometric fidelity after a few feedback iterations, making BIM creation more accessible.
Building Information Modeling (BIM) is a cornerstone technology in the Architecture, Engineering, and Construction (AEC) industry, providing detailed, data-rich digital representations of buildings. However, creating these BIM models from scratch is often a complex and time-consuming task, demanding significant expertise in specialized software like Autodesk Revit. This complexity creates a barrier for beginners, students, and those in the early design stages, especially when working with informal inputs like hand-drawn sketches.
Traditional methods for converting 2D floor plans to 3D BIM models often struggle with unscaled, irregular, or abstract hand-drawn sketches. They typically require clean, scaled, and often “Manhattan-style” inputs, meaning layouts where walls are strictly horizontal or vertical. This limitation makes it difficult to handle designs with angled, curved, or free-form geometries, which are common in early conceptual sketches.
Introducing Sketch2BIM: Bridging Sketches to 3D Models
A new research paper introduces “Sketch2BIM,” a groundbreaking human-in-the-loop pipeline designed to transform unscaled, hand-drawn floor plan sketches into semantically consistent 3D BIM models. This innovative workflow leverages the power of multimodal large language models (MLLMs) within a multi-agent framework, combining automated perception with crucial human feedback and validation. The goal is to make BIM creation accessible to both experts and non-experts using just freehand sketches. You can read the full research paper here: Sketch2BIM: A Multi-Agent Human-AI Collaborative Pipeline to Convert Hand-Drawn Floor Plans to 3D BIM.
How Sketch2BIM Works: A Two-Phase Approach
The Sketch2BIM pipeline operates in two main phases, orchestrated by several specialized AI agents, with OpenAI’s GPT-5 serving as the core MLLM for its multimodal capabilities and reasoning power.
Phase 1: Layout Extraction (Sketch to JSON)
This initial phase focuses on converting a raw, hand-drawn sketch into a structured 2D JSON (JavaScript Object Notation) layout. This JSON file precisely defines elements like walls, doors, and windows.
- Perceptual Extraction (Agent A1): This agent processes the hand-drawn image, interpreting visual geometry, line structures, and any textual annotations. It identifies walls, doors, and windows, even handling complex “non-Manhattan” layouts with angled or curved walls. It also estimates global skew and pixel-to-foot scaling from annotations.
- Human Feedback Integration (Agent A2): This is where the “human-in-the-loop” comes in. Users can review the initial JSON output and provide natural language feedback (e.g., “move door 3 to wall 8,” “extend wall 7 downward by 8 feet”). Agent A2 then refines the layout based on this feedback.
- Schema and Topology Validation (Agent A3): This agent acts as a quality control, ensuring the JSON output adheres to strict architectural and geometric rules. It checks for correct syntax, unique IDs, consistent lengths, and proper connectivity of elements.
- Orchestration Controller (Agent A4): This agent supervises the entire Phase 1 process, managing memory, feedback history, and guiding the iterative refinement until both human approval and validation checks pass.
This iterative process ensures that the final 2D JSON layout is topologically sound and semantically consistent, accurately representing the hand-drawn sketch.
Phase 2: Revit Script Generation and Validation
Once the finalized 2D JSON layout is ready, Phase 2 takes over to create the 3D BIM model in Autodesk Revit, a widely used BIM platform.
- Script Synthesis (Agent A5): This agent translates each element from the JSON file into specific commands for the Revit API (Application Programming Interface). It ensures that elements like walls, doors, windows, and floors are created in the correct order and with appropriate properties.
- Runtime Error Correction (Agent A6): Before and during execution, a validation module checks the generated Python script for structural code constraints. If any runtime errors or logical faults occur during the script’s execution in RevitPythonShell (RPS), Agent A6 identifies them and iteratively applies targeted code edits until the script runs successfully.
The outcome of Phase 2 is a procedurally accurate and semantically consistent 3D BIM model within Revit, faithfully translating the hand-drawn sketch into a digital building representation.
Demonstrated Performance and Accuracy
The Sketch2BIM pipeline was rigorously evaluated on ten diverse hand-drawn floor plans, specifically designed to challenge the system with various complexities, including Manhattan and non-Manhattan geometries, concavities, skew, and curved walls. The evaluation focused heavily on Phase 1, as errors here directly impact the final BIM model.
- High Detection Quality: Metrics like precision, recall, and F1 scores were used to assess how accurately elements (walls, doors, windows) were detected. Doors were captured with high reliability in the initial pass, often achieving perfect scores. Wall detection started strong (around 83% recall) and quickly achieved near-perfect alignment after a few feedback iterations. Windows also showed rapid improvement with feedback.
- Strong Geometric Fidelity: The accuracy of element dimensions and placements was measured using Root Mean Square Error (RMSE) for length errors and Mean Absolute Error (MAE) for positional errors. Initially, walls showed the largest length errors, while windows had the largest positional errors. However, with human feedback, both RMSE and MAE progressively decreased to zero across all categories, demonstrating that the system effectively corrects sizing and alignment issues.
- Successful 3D BIM Generation: Human reviewers, including graduate students with BIM expertise, validated the 3D BIM models generated in Revit. They confirmed that walls were correctly positioned, openings were integrated properly, connections were intact, and angled/curved geometries were preserved. A single continuous floor slab was also correctly generated to cover the building footprint.
These results highlight that the human-in-the-loop approach, combined with the multi-agent framework, allows for rapid convergence to accurate and usable BIM models, even from imperfect hand-drawn inputs.
Also Read:
- Bridging the Gap: A Multi-Agent AI System Translates Natural Language into Spatial SQL
- DeepAgent: Advancing AI with Autonomous Reasoning and Dynamic Tool Use
Current Limitations and Future Potential
While Sketch2BIM represents a significant leap forward, the current implementation has some limitations. It primarily focuses on core structural components like walls, doors, windows, and slabs, and does not yet generate more complex architectural features such as staircases, curtains, multi-story components, or Mechanical, Electrical, and Plumbing (MEP) elements. For non-Manhattan layouts, it generates a single continuous floor slab, which ensures coverage but limits flexibility. The system’s performance is also tied to the stability of the underlying MLLM (GPT-5), and the evaluation was based on a limited dataset of ten sketches.
Despite these limitations, the Sketch2BIM pipeline holds immense potential to democratize BIM creation. By enabling anyone who can sketch a floor plan to generate accurate 3D models without extensive BIM expertise or model training, it significantly lowers the technical barrier to adopting advanced modeling tools in the AEC industry.
