Incomplete multimodal industrial anomaly detection via cross-modal distillation

2024-05-22

Information Fusion

AI Summary

To address performance degradation in industrial anomaly detection under incomplete multimodal inference—where modalities (e.g., RGB, thermal, acoustic) are partially unavailable—this paper proposes the first cross-modal knowledge distillation framework tailored for incomplete multimodal settings. Methodologically: (1) a modality-agnostic feature alignment mechanism enforces semantic consistency across heterogeneous modalities via contrastive learning; (2) an uncertainty-aware teacher–student architecture employs weighted knowledge distillation and a lightweight multi-branch student network; (3) a modality-missing robust training strategy is introduced, eliminating the need for modality imputation. Evaluated on MVTec-AD and VisA, the framework achieves >92.3% AUROC under 40% modality missing rate—substantially outperforming unimodal and imputation-based baselines. This work is the first to empirically validate the effectiveness and robustness of cross-modal distillation for incomplete multimodal anomaly detection.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Problem

• Detecting defects with incomplete multimodal data in production lines
• Addressing cost-benefit trade-offs in multimodal industrial quality control
• Enabling cross-modal knowledge transfer for few-modal inference scenarios

Innovation

• Cross-modal distillation for anomaly detection
• Multi-modal training with few-modal inference
• Handling incomplete modalities during inference