Vision-Ultrasound Robotic System based on Deep Learning for Gas and Arc Hazard Detection in Manufacturing

2025-02-08

AI Summary

To address the critical challenge of detecting hazardous gas leaks and arc discharges in industrial manufacturing environments, this paper proposes a vision–ultrasound dual-modal autonomous inspection robot system enabling real-time, edge-deployed detection and classification. The method innovatively integrates vision-guided localization with beamforming-enhanced ultrasonic analysis, introduces a Gamma correction–STFT feature enhancement technique, and establishes a trustworthy multi-source heterogeneous data fusion framework coupled with a human–machine collaborative detection paradigm. Implemented on an embedded platform, the system performs real-time beamforming and Inception-inspired CNN inference using a 112-channel, 96-kHz acoustic camera. Experimental results demonstrate a 99% detection accuracy for gas leaks—44 percentage points higher than conventional methods under strong reverberation and noise—and achieve end-to-end onboard inference in only 2.1 seconds.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Gas leaks and arc discharges present significant risks in industrial environments, requiring robust detection systems to ensure safety and operational efficiency. Inspired by human protocols that combine visual identification with acoustic verification, this study proposes a deep learning-based robotic system for autonomously detecting and classifying gas leaks and arc discharges in manufacturing settings. The system is designed to execute all experimental tasks entirely onboard the robot. Utilizing a 112-channel acoustic camera operating at a 96 kHz sampling rate to capture ultrasonic frequencies, the system processes real-world datasets recorded in diverse industrial scenarios. These datasets include multiple gas leak configurations (e.g., pinhole, open end) and partial discharge types (Corona, Surface, Floating) under varying environmental noise conditions. Proposed system integrates visual detection and a beamforming-enhanced acoustic analysis pipeline. Signals are transformed using STFT and refined through Gamma Correction, enabling robust feature extraction. An Inception-inspired CNN further classifies hazards, achieving 99% gas leak detection accuracy. The system not only detects individual hazard sources but also enhances classification reliability by fusing multi-modal data from both vision and acoustic sensors. When tested in reverberation and noise-augmented environments, the system outperformed conventional models by up to 44%p, with experimental tasks meticulously designed to ensure fairness and reproducibility. Additionally, the system is optimized for real-time deployment, maintaining an inference time of 2.1 seconds on a mobile robotic platform. By emulating human-like inspection protocols and integrating vision with acoustic modalities, this study presents an effective solution for industrial automation, significantly improving safety and operational reliability.

Problem

• Detects gas leaks and arc discharges
• Uses deep learning for hazard classification
• Integrates vision and acoustic sensors

Innovation

• Deep learning-based robotic system
• Vision-acoustic data fusion
• Real-time industrial hazard detection