Depth Matters: Exploring Deep Interactions of RGB-D for Semantic Segmentation in Traffic Scenes

2024-09-12

arXiv.org

AI Summary

Existing RGB-D semantic segmentation methods neglect the intrinsic spatial-geometric properties of depth maps, leading to attention misalignment and feature mismatch. To address this, we propose DiPFormer—a learnable Depth-interaction Pyramid Transformer—specifically designed for traffic scenes. DiPFormer is the first Transformer architecture to deeply fuse depth-based geometric priors with RGB semantic features. It introduces a Depth Spatial-Aware Optimization (Depth SAO) bias modeling module to explicitly encode real-world spatial relationships; employs Depth Linear Cross-Attention (Depth LCA) for pixel-level RGB-D feature alignment; and integrates a multi-scale MLP decoder with a dual-stream interaction mechanism. Evaluated on KITTI Road, KITTI-360, and Cityscapes, DiPFormer achieves absolute improvements of +7.5%, +4.9%, and 83.4% in mIoU, respectively, establishing new state-of-the-art performance at the time of publication.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance

Abstract

RGB-D has gradually become a crucial data source for understanding complex scenes in assisted driving. However, existing studies have paid insufficient attention to the intrinsic spatial properties of depth maps. This oversight significantly impacts the attention representation, leading to prediction errors caused by attention shift issues. To this end, we propose a novel learnable Depth interaction Pyramid Transformer (DiPFormer) to explore the effectiveness of depth. Firstly, we introduce Depth Spatial-Aware Optimization (Depth SAO) as offset to represent real-world spatial relationships. Secondly, the similarity in the feature space of RGB-D is learned by Depth Linear Cross-Attention (Depth LCA) to clarify spatial differences at the pixel level. Finally, an MLP Decoder is utilized to effectively fuse multi-scale features for meeting real-time requirements. Comprehensive experiments demonstrate that the proposed DiPFormer significantly addresses the issue of attention misalignment in both road detection (+7.5%) and semantic segmentation (+4.9% / +1.5%) tasks. DiPFormer achieves state-of-the-art performance on the KITTI (97.57% F-score on KITTI road and 68.74% mIoU on KITTI-360) and Cityscapes (83.4% mIoU) datasets.

Problem

• Addresses attention misalignment in RGB-D semantic segmentation
• Explores spatial properties of depth maps for scene understanding
• Improves road detection and segmentation accuracy in traffic scenes

Innovation

• Depth Spatial-Aware Optimization for spatial relationships
• Depth Linear Cross-Attention for pixel-level differences
• MLP Decoder for multi-scale feature fusion