An Integrated Hardware-Software Design for Low-Data Spatial Defect Detection in Robotic Visual Inspection with Hybrid Optoelectronic Neural Networks

2026-06-23

AI Summary

This work addresses the challenges of high data redundancy and costly defect annotation in robotic visual inspection by proposing a hardware-software co-designed optoelectronic neural network architecture. Leveraging a non-imaging, low-data paradigm, the approach achieves intrinsic data reduction and eliminates the need for fine-grained annotations by integrating a digital micromirror device (DMD) to implement an in-sensor optical convolution layer that performs feature extraction and block-wise compressive sensing directly in the optical domain. A CLIP-guided attention mechanism enables natural language-driven defect localization, with localization accuracy quantified via a newly introduced LAA metric. Evaluated on transparent material defect detection, the method reduces data volume by 90% and cuts CNN computational load by 60% compared to conventional imaging-based approaches, while maintaining comparable detection accuracy.

Abstract

To address data overload and inefficient shape-level annotation in robotic visual inspection, this paper proposes a hardware-software integrated optoelectronic architecture. A non-imaging, low-data paradigm is established to minimize annotation dependency. First, a sensor-in-the-loop strategy reconfigures a Digital Micromirror Device (DMD) as a physical optical convolutional layer, enabling photonic-domain feature extraction that unifies sensing hardware and processing software. To suppress data volume at the source, a block-based compressed sensing strategy encodes spatial information into low-dimensional temporal signals, drastically reducing redundancy. Subsequently, to bypass laborious manual defect shape annotation, natural language descriptions guide the network to align with highly generalizable features from Contrastive Language-Image Pre-training (CLIP), steering the attention maps of the optoelectronic neural network toward defect shapes. Furthermore, a Localization Accuracy for Attention (LAA) metric is proposed to quantify shape-level defect localization performance. Experiments on transparent material defect detection validate the system's effectiveness. Parametric analysis reveals how measurement matrices, compression ratios, and block sizes affect accuracy. Results show that, compared to traditional imaging, the proposed architecture maintains equivalent accuracy while reducing data volume by 90% for Vision Transformers and computational workload by 60% for Convolutional Neural Networks. This low-data paradigm offers an efficient solution for industrial automation scenarios involving massive data streams, high acquisition costs, or constrained edge resources.

Problem

• data overload
• shape-level annotation
• robotic visual inspection
• low-data
• defect detection

Innovation

• optoelectronic neural networks
• sensor-in-the-loop
• compressed sensing
• language-guided defect detection
• low-data paradigm