Ortho2CAD: 3D CAD generation from orthographic drawings using vision language models

2026-07-09

AI Summary

This study addresses the challenge of reconstructing editable, parameterized 3D CAD models from engineering orthographic views. The work proposes the first end-to-end framework that integrates a vision-language model (VLM), supervised fine-tuning (SFT), and geometry-guided reinforcement learning (RL) to directly generate syntactically correct CadQuery code. By constructing a large-scale synthetic dataset supporting hidden lines and dimension annotations, and leveraging pythonOCC with STEP-based rendering for training, the method achieves 100% syntactic validity in generated code across multiple benchmarks. It further demonstrates an average IoU improvement of over 7% relative to existing approaches, significantly outperforming current state-of-the-art techniques.

Abstract

Engineering design intent is often communicated through rasterized orthographic drawings. However, downstream workflows inherently require editable and parametrically defined 3D computer-aided design (CAD) models. To bridge this gap, we introduce Ortho2CAD, a vision-language model (VLM) specifically designed to translate rasterized orthographic drawings directly into editable CadQuery code, which can then be seamlessly converted into 3D CAD models. To train the model effectively, we utilize supervised fine-tuning (SFT) for instances where explicit CadQuery code labels already exist, and we apply geometry-grounded reinforcement learning (RL) to optimize the model in scenarios where ground-truth labels are absent. To enable learning at scale, we create a pythonOCC-based drawing generator that renders first-angle orthographic projections from STEP models, complete with dashed hidden lines and key dimensions. On existing datasets encompassing settings both with and without CadQuery supervision, we generate orthographic drawings and show that our model produces 100% syntactically valid code. Moreover, it achieves a 3D CAD intersection-over-union (IoU) accuracy that surpasses all baselines, with an average relative improvement of over 7% compared directly against the next best performing model. We show that leveraging VLMs with SFT and RL techniques can effectively pave the way forward for orthographic drawing to 3D CAD reconstruction. Our implementation is available at https://github.com/AdityaJoglekar/Ortho2CAD.

Problem

• orthographic drawings
• 3D CAD generation
• design intent
• editable CAD models
• rasterized engineering drawings

Innovation

• vision-language model
• orthographic drawing
• 3D CAD generation
• reinforcement learning
• CadQuery