Geo-ConvGRU: Geographically Masked Convolutional Gated Recurrent Unit for Bird-Eye View Segmentation

2024-12-28

AI Summary

To address the limitations of 3D CNNs in long-range temporal modeling and the spatiotemporal imbalance and high computational cost of Transformers in bird’s-eye-view (BEV) semantic segmentation, this paper proposes a lightweight and efficient joint spatiotemporal modeling framework. The core innovation is a learnable geo-aware masking mechanism embedded within the convolutional gating structure of ConvGRU, which explicitly suppresses temporal noise from irrelevant regions while preserving the local receptive field of CNNs and the sequential modeling capability of RNNs. Evaluated on the NuScenes dataset, the method achieves state-of-the-art performance in BEV semantic segmentation, with significant improvements in segmentation accuracy. Moreover, it reduces model parameters by approximately 40% and accelerates inference speed by 2.1× compared to baseline approaches—outperforming mainstream Transformer-based methods in both efficiency and accuracy.

Technology Category

• Computer Vision: Segmentation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Convolutional Neural Networks (CNNs) have significantly impacted various computer vision tasks, however, they inherently struggle to model long-range dependencies explicitly due to the localized nature of convolution operations. Although Transformers have addressed limitations in long-range dependencies for the spatial dimension, the temporal dimension remains underexplored. In this paper, we first highlight that 3D CNNs exhibit limitations in capturing long-range temporal dependencies. Though Transformers mitigate spatial dimension issues, they result in a considerable increase in parameter and processing speed reduction. To overcome these challenges, we introduce a simple yet effective module, Geographically Masked Convolutional Gated Recurrent Unit (Geo-ConvGRU), tailored for Bird's-Eye View segmentation. Specifically, we substitute the 3D CNN layers with ConvGRU in the temporal module to bolster the capacity of networks for handling temporal dependencies. Additionally, we integrate a geographical mask into the Convolutional Gated Recurrent Unit to suppress noise introduced by the temporal module. Comprehensive experiments conducted on the NuScenes dataset substantiate the merits of the proposed Geo-ConvGRU, revealing that our approach attains state-of-the-art performance in Bird's-Eye View segmentation.

Problem

• 3D CNN
• Transformer
• Bird's Eye View Image Segmentation

Innovation

• Geo-ConvGRU
• Bird's Eye View Segmentation
• NuScenes Dataset