Mask-adaptive Gated Convolution and Bi-directional Progressive Fusion Network for Depth Completion

2024-01-15

arXiv.org

AI Summary

To address depth map completion in sparse depth images with missing pixels, this paper proposes an RGB-guided depth completion network. Methodologically, we design a mask-adaptive gated convolution (MagaConv) to enable missing-region-aware dynamic feature modulation, and introduce a bidirectional progressive fusion (BP-Fusion) module to jointly integrate depth and RGB features across a multi-scale feature pyramid. The network is trained end-to-end using a CNN architecture augmented with a multi-stage progressive upsampling strategy. Evaluated on NYU-Depth V2, DIML, and SUN RGB-D benchmarks, our approach achieves state-of-the-art performance, significantly outperforming existing methods in RMSE, REL, and other key metrics. It enhances completion accuracy, robustness to occlusions and sensor noise, and structural fidelity of recovered depth maps.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Depth completion is a critical task for handling depth images with missing pixels, which can negatively impact further applications. Recent approaches have utilized Convolutional Neural Networks (CNNs) to reconstruct depth images with the assistance of color images. However, vanilla convolution has non-negligible drawbacks in handling missing pixels. To solve this problem, we propose a new model for depth completion based on an encoder-decoder structure. Our model introduces two key components: the Mask-adaptive Gated Convolution (MagaConv) architecture and the Bi-directional Progressive Fusion (BP-Fusion) module. The MagaConv architecture is designed to acquire precise depth features by modulating convolution operations with iteratively updated masks, while the BP-Fusion module progressively integrates depth and color features, utilizing consecutive bi-directional fusion structures in a global perspective. Extensive experiments on popular benchmarks, including NYU-Depth V2, DIML, and SUN RGB-D, demonstrate the superiority of our model over state-of-the-art methods. We achieved remarkable performance in completing depth maps and outperformed existing approaches in terms of accuracy and reliability.

Problem

• Depth Image
• Pixel Deficiency
• Image Completion

Innovation

• MagaConv
• BP-Fusion
• Depth Completion