Glass Surface Segmentation with an RGB-D Camera via Weighted Feature Fusion for Service Robots

2025-08-03

AI Summary

Service robots face significant challenges in segmenting glass surfaces using RGB-D cameras in real-world scenarios due to glass transparency, strong reflections, and occlusions. To address these issues, this paper proposes a Weighted Feature Fusion (WFF) module that enables dynamic, adaptive fusion of RGB and depth features; the module is plug-and-play and compatible with multiple mainstream segmentation backbones. Furthermore, we introduce MJU-Glass—the first real-world glass segmentation dataset collected *in situ* by service robots—filling a critical gap in publicly available glass segmentation data. Integrating WFF into architectures such as PSPNet yields substantial improvements in segmentation robustness without compromising computational efficiency: boundary IoU increases by 7.49%, and mean IoU also improves significantly, thereby effectively reducing robot collision risk during navigation and interaction.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Natural Language Processing: Safety and Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

We address the problem of glass surface segmentation with an RGB-D camera, with a focus on effectively fusing RGB and depth information. To this end, we propose a Weighted Feature Fusion (WFF) module that dynamically and adaptively combines RGB and depth features to tackle issues such as transparency, reflections, and occlusions. This module can be seamlessly integrated with various deep neural network backbones as a plug-and-play solution. Additionally, we introduce the MJU-Glass dataset, a comprehensive RGB-D dataset collected by a service robot navigating real-world environments, providing a valuable benchmark for evaluating segmentation models. Experimental results show significant improvements in segmentation accuracy and robustness, with the WFF module enhancing performance in both mean Intersection over Union (mIoU) and boundary IoU (bIoU), achieving a 7.49% improvement in bIoU when integrated with PSPNet. The proposed module and dataset provide a robust framework for advancing glass surface segmentation in robotics and reducing the risk of collisions with glass objects.

Problem

• Segment glass surfaces using RGB-D camera data
• Fuse RGB and depth features adaptively for transparency issues
• Improve segmentation accuracy and reduce robot collisions

Innovation

• Dynamic RGB-D fusion via Weighted Feature Fusion
• Plug-and-play module for deep neural networks
• Comprehensive MJU-Glass dataset for benchmarking