TOOLCAD: Exploring Tool-Using Large Language Models in Text-to-CAD Generation with Reinforcement Learning

2026-04-09

AI Summary

Existing approaches struggle to effectively orchestrate large language models (LLMs) in interacting with CAD engines, hindering autonomous text-to-3D-CAD generation. To address this challenge, this work proposes ToolCAD, a novel framework that establishes the first agent-based training paradigm tailored for CAD tasks. ToolCAD leverages an interactive modeling environment, a hybrid feedback mechanism, and curriculum-based reinforcement learning to enhance LLMs’ geometric reasoning capabilities. Additionally, it introduces a CAD-specific chain-of-thought (CAD-CoT) prompting strategy to improve planning accuracy. Experimental results demonstrate that the proposed method significantly boosts the performance of open-source LLMs on specialized CAD tasks, enabling trained agents to achieve performance on par with proprietary models. This advancement paves the way toward highly usable and robust autonomous systems for text-to-CAD generation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Computer-Aided Design (CAD) is an expert-level task that relies on long-horizon reasoning and coherent modeling actions. Large Language Models (LLMs) have shown remarkable advancements in enabling language agents to tackle real-world tasks. Notably, there has been no investigation into how tool-using LLMs optimally interact with CAD engines, hindering the emergence of LLM-based agentic text-to-CAD modeling systems. We propose ToolCAD, a novel agentic CAD framework deploying LLMs as tool-using agents for text-to-CAD generation. Furthermore, we introduce an interactive CAD modeling gym to rollout reasoning and tool-augmented interaction trajectories with the CAD engine, incorporating hybrid feedback and human supervision. Meanwhile, an end-to-end post-training strategy is presented to enable the LLM agent to elicit refined CAD Modeling Chain of Thought (CAD-CoT) and evolve into proficient CAD tool-using agents via online curriculum reinforcement learning. Our findings demonstrate ToolCAD fills the gap in adopting and training open-source LLMs for CAD tool-using agents, enabling them to perform comparably to proprietary models, paving the way for more accessible and robust autonomous text-to-CAD modeling systems.

Problem

• Tool-using LLMs
• Text-to-CAD Generation
• CAD Modeling
• Language Agents
• Reinforcement Learning

Innovation

• tool-using LLMs
• text-to-CAD generation
• reinforcement learning
• CAD Modeling Chain of Thought
• interactive CAD gym