CAD-Prompted SAM3: Geometry-Conditioned Instance Segmentation for Industrial Objects

2026-02-24

AI Summary

This work addresses the challenge of accurately segmenting geometrically complex and materially diverse industrial objects, where linguistic or appearance-based prompts often fail. To overcome this limitation, the authors propose a geometry-conditioned prompting mechanism that leverages CAD models by introducing multi-view rendered images as geometric priors into the promptable segmentation framework SAM3. This approach enables appearance-agnostic, single-stage instance segmentation. By combining synthetic data training with multi-view CAD renderings, the method significantly improves segmentation accuracy and robustness across a wide range of materials and lighting conditions, effectively supporting industrial object segmentation tasks that cannot be reliably described by language or visual appearance alone.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Prompt Engineering / Prompting
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Verbal-prompted segmentation is inherently limited by the expressiveness of natural language and struggles with uncommon, instance-specific, or difficult-to-describe objects: scenarios frequently encountered in manufacturing and 3D printing environments. While image exemplars provide an alternative, they primarily encode appearance cues such as color and texture, which are often unrelated to a part's geometric identity. In industrial settings, a single component may be produced in different materials, finishes, or colors, making appearance-based prompting unreliable. In contrast, such objects are typically defined by precise CAD models that capture their canonical geometry. We propose a CAD-prompted segmentation framework built on SAM3 that uses canonical multi-view renderings of a CAD model as prompt input. The rendered views provide geometry-based conditioning independent of surface appearance. The model is trained using synthetic data generated from mesh renderings in simulation under diverse viewpoints and scene contexts. Our approach enables single-stage, CAD-prompted mask prediction, extending promptable segmentation to objects that cannot be robustly described by language or appearance alone.

Problem

• instance segmentation
• industrial objects
• CAD models
• geometry-conditioned
• promptable segmentation

Innovation

• CAD-prompted segmentation
• geometry-conditioned
• instance segmentation
• multi-view rendering
• SAM3