From Idea to CAD: A Language Model-Driven Multi-Agent System for Collaborative Design

2025-03-06

AI Summary

Industrial CAD modeling suffers from low efficiency due to its reliance on multi-role collaboration and deep domain expertise. To address this, we propose the first Vision-Language Model (VLM)-driven multi-agent system tailored for CAD design, decoupling requirements analysis, parametric modeling, and visual quality inspection into specialized, collaborative agent roles. Our system enables end-to-end generation of editable CAD models directly from hand-drawn sketches or textual specifications. It innovatively integrates a vision-language model, parametric CAD APIs, tool-documentation-enhanced reasoning, and a visual quality assessment module, while supporting user-feedback-driven closed-loop refinement. We validate system usability across industrial and maker scenarios. Ablation studies demonstrate that each agent module significantly improves modeling accuracy (+23.6%) and user satisfaction (+31.4%).

Technology Category

• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Creating digital models using Computer Aided Design (CAD) is a process that requires in-depth expertise. In industrial product development, this process typically involves entire teams of engineers, spanning requirements engineering, CAD itself, and quality assurance. We present an approach that mirrors this team structure with a Vision Language Model (VLM)-based Multi Agent System, with access to parametric CAD tooling and tool documentation. Combining agents for requirements engineering, CAD engineering, and vision-based quality assurance, a model is generated automatically from sketches and/ or textual descriptions. The resulting model can be refined collaboratively in an iterative validation loop with the user. Our approach has the potential to increase the effectiveness of design processes, both for industry experts and for hobbyists who create models for 3D printing. We demonstrate the potential of the architecture at the example of various design tasks and provide several ablations that show the benefits of the architecture's individual components.

Problem

• Automates CAD model creation from sketches or text descriptions.
• Enhances design process efficiency for experts and hobbyists.
• Integrates multi-agent system for collaborative, iterative design refinement.

Innovation

• VLM-based Multi Agent System for CAD
• Automated model generation from sketches/text
• Iterative validation loop with user collaboration