QueryCAD: Grounded Question Answering for CAD Models

2024-09-13

arXiv.org

AI Summary

Industrial CAD models suffer from insufficient semantic understanding, hindering automated robot program synthesis. Method: We propose the first natural language question-answering framework for CAD models, comprising (1) SegCAD—an open-vocabulary 3D CAD instance segmentation model enabling geometry-semantic alignment at the component level; (2) a multimodal prompt engineering strategy with vision-language joint modeling; and (3) CAD-QA—the first benchmark dataset for CAD question answering. Results: Our method significantly outperforms existing baselines on CAD-QA and has been successfully integrated into a robot program synthesis system, substantially improving the accuracy of CAD-driven automated reasoning. The core contribution is the first demonstration of precise, natural-language-driven extraction of structured information from CAD models, effectively bridging the critical gap between CAD semantic understanding and downstream robotic tasks.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Segmentation

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

CAD models are widely used in industry and are essential for robotic automation processes. However, these models are rarely considered in novel AI-based approaches, such as the automatic synthesis of robot programs, as there are no readily available methods that would allow CAD models to be incorporated for the analysis, interpretation, or extraction of information. To address these limitations, we propose QueryCAD, the first system designed for CAD question answering, enabling the extraction of precise information from CAD models using natural language queries. QueryCAD incorporates SegCAD, an open-vocabulary instance segmentation model we developed to identify and select specific parts of the CAD model based on part descriptions. We further propose a CAD question answering benchmark to evaluate QueryCAD and establish a foundation for future research. Lastly, we integrate QueryCAD within an automatic robot program synthesis framework, validating its ability to enhance deep-learning solutions for robotics by enabling them to process CAD models (https://claudius-kienle.github.com/querycad).

Problem

• Lack of AI methods for CAD model analysis and information extraction.
• Need for natural language-based querying of CAD models.
• Integration of CAD models into robotic automation processes.

Innovation

• QueryCAD enables CAD model information extraction via natural language queries
• SegCAD identifies CAD parts using open-vocabulary instance segmentation
• QueryCAD integrates with robot program synthesis for enhanced automation