RADAR: Robust Two-stage Modality-incomplete Industrial Anomaly Detection

2024-10-02

arXiv.org

AI Summary

In industrial quality inspection, multimodal anomaly detection (MIAD) models suffer from poor robustness and overfitting due to frequent modality incompleteness—i.e., missing 2D RGB images or 3D point clouds. To address this, we formally define the Modality-Incomplete Industrial Anomaly Detection (MIIAD) task—the first systematic formulation of MIAD under partial modality availability. We propose RADAR, a two-stage robust framework: (1) an instruction-guided adaptive cross-modal fusion mechanism that dynamically adjusts to missing modalities; and (2) a real-pseudo modality hybrid detection module integrating pretrained multimodal Transformers, HyperNetwork-driven dynamic parameter generation, and modality-missing instruction fine-tuning. Evaluated on our newly constructed MIIAD benchmark, RADAR consistently outperforms state-of-the-art methods across diverse incomplete-modality scenarios—including single- and dual-modality absence—achieving superior detection accuracy and strong generalization robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Multimodal Industrial Anomaly Detection (MIAD), utilizing 3D point clouds and 2D RGB images to identify the abnormal region of products, plays a crucial role in industrial quality inspection. However, the conventional MIAD setting presupposes that all 2D and 3D modalities are paired, overlooking the fact that multimodal data collected from the real world is often imperfect due to missing modalities. Consequently, MIAD models that demonstrate robustness against modal-incomplete data are highly desirable in practice. To address this practical challenge, we introduce a first-of-its-kind study that comprehensively investigates Modality-Incomplete Industrial Anomaly Detection (MIIAD), to consider the imperfect learning environment in which the multimodal information may be incomplete. Not surprisingly, we discovered that most existing MIAD approaches are inadequate for addressing MIIAD challenges, leading to significant performance degradation on the MIIAD benchmark we developed. In this paper, we propose a novel two-stage Robust modAlity-imcomplete fusing and Detecting frAmewoRk, abbreviated as RADAR. Our bootstrapping philosophy is to enhance two stages in MIIAD, improving the robustness of the Multimodal Transformer: i) In feature fusion, we first explore learning modality-incomplete instruction, guiding the pre-trained Multimodal Transformer to robustly adapt to various modality-incomplete scenarios, and implement adaptive parameter learning based on a HyperNetwork; ii) In anomaly detection, we construct a real-pseudo hybrid module to highlight the distinctiveness of modality combinations, further enhancing the robustness of the MIIAD model. Our experimental results demonstrate that the proposed RADAR significantly surpasses conventional MIAD methods in terms of effectiveness and robustness on our newly created MIIAD dataset, underscoring its practical application value.

Problem

• Detecting anomalies with incomplete multimodal industrial data
• Addressing performance degradation from missing 2D/3D modalities
• Preventing model overfitting in modality-incomplete scenarios

Innovation

• Modality-incomplete Instruction guides multimodal Transformer adaptation
• HyperNetwork enables adaptive parameter learning for robustness
• Double-Pseudo Hybrid Module mitigates overfitting in incomplete modalities