SiM3D: Single-instance Multiview Multimodal and Multisetup 3D Anomaly Detection Benchmark

2025-06-26

AI Summary

This work addresses the challenging task of single-instance 3D anomaly detection and segmentation (ADS) in manufacturing—where models must be trained on only one real or synthetic instance and generalize to unseen real-world scenarios. Method: We introduce the first multi-view, multimodal, multi-station benchmark tailored to this setting, integrating 12-Megapixel RGB images, 7-Million-point 3D point clouds, and CAD models. Our approach features CAD-guided voxel-level 3D annotation, a mechanism to adapt single-view methods to multi-view voxel space, and an anomaly-volume-based cross-domain evaluation paradigm. Contribution/Results: We release a high-quality multimodal dataset comprising 333 instances across eight industrial part categories, with fine-grained 3D segmentation ground truth. We establish reproducible baselines and empirically demonstrate that multi-view fusion substantially improves cross-domain generalization—from synthetic to real—under the single-instance regime.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose SiM3D, the first benchmark considering the integration of multiview and multimodal information for comprehensive 3D anomaly detection and segmentation (ADS), where the task is to produce a voxel-based Anomaly Volume. Moreover, SiM3D focuses on a scenario of high interest in manufacturing: single-instance anomaly detection, where only one object, either real or synthetic, is available for training. In this respect, SiM3D stands out as the first ADS benchmark that addresses the challenge of generalising from synthetic training data to real test data. SiM3D includes a novel multimodal multiview dataset acquired using top-tier industrial sensors and robots. The dataset features multiview high-resolution images (12 Mpx) and point clouds (7M points) for 333 instances of eight types of objects, alongside a CAD model for each type. We also provide manually annotated 3D segmentation GTs for anomalous test samples. To establish reference baselines for the proposed multiview 3D ADS task, we adapt prominent singleview methods and assess their performance using novel metrics that operate on Anomaly Volumes.

Problem

• Develops first benchmark for multiview multimodal 3D anomaly detection
• Addresses single-instance training with synthetic-to-real generalization challenge
• Introduces high-resolution industrial dataset with annotated 3D anomalies

Innovation

• Multiview multimodal 3D anomaly detection benchmark
• Single-instance training with synthetic data
• High-resolution images and point clouds dataset