Generalizable Autonomous Driving System across Diverse Adverse Weather Conditions

2024-09-23

AI Summary

To address the poor generalization of autonomous driving semantic segmentation under adverse weather conditions (e.g., rain, fog, snow, and their combinations) and its heavy reliance on clean-weather annotated data, this paper proposes AdvImmu—the first reference-image-free, weather-immune cross-weather segmentation framework. Methodologically, we design a local temporal modeling module coupled with a global shuffling mechanism to extract short-term weather-invariant features, and introduce an unfolding regularizer to enhance robustness. Furthermore, we leverage the Segment Anything Model (SAM) jointly with our novel SBICAC clustering algorithm to generate high-quality pseudo-labels, significantly alleviating annotation dependency. On cross-weather semantic segmentation benchmarks, AdvImmu achieves an mIoU improvement of 88.56% over state-of-the-art methods, demonstrating substantial gains in generalization to unseen adverse weather combinations and overall robustness.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Various adverse weather conditions pose a significant challenge to autonomous driving (AD) street scene semantic understanding (segmentation). A common strategy is to minimize the disparity between images captured in clear and adverse weather conditions. However, this technique typically relies on utilizing clear image as a reference, which is challenging to obtain in practice. Furthermore, this method typically targets a single adverse condition, and thus perform poorly when confronting a mixture of multiple adverse weather conditions. To address these issues, we introduce a reference-free and Adverse weather-Immune scheme (called AdvImmu) that leverages the invariance of weather conditions over short periods (seconds). Specifically, AdvImmu includes three components: Locally Sequential Mechanism (LSM), Globally Shuffled Mechanism (GSM), and Unfolded Regularizers (URs). LSM leverages temporal correlations between adjacent frames to enhance model performance. GSM is proposed to shuffle LSM segments to prevent overfitting of temporal patterns. URs are the deep unfolding implementation of two proposed regularizers to penalize the model complexity to enhance across-weather generalization. In addition, to overcome the over-reliance on consecutive frame-wise annotations in the training of AdvImmu (typically unavailable in AD scenarios), we incorporate a foundation model named Segment Anything Model (SAM) to assist to annotate frames, and additionally propose a cluster algorithm (denoted as SBICAC) to surmount SAM's category-agnostic issue to generate pseudo-labels. Extensive experiments demonstrate that the proposed AdvImmu outperforms existing state-of-the-art methods by 88.56% in mean Intersection over Union (mIoU).

Problem

• Autonomous Driving
• Adverse Weather Conditions
• Scene Recognition

Innovation

• AdvImmu
• Weather-Resilient Recognition
• LSM_GSM_URs