WeatherDG: LLM-assisted Procedural Weather Generation for Domain-Generalized Semantic Segmentation

2024-10-15

arXiv.org

AI Summary

Weak generalization of semantic segmentation models across weather domains (e.g., rain, fog, snow) and poor robustness to long-tailed classes (e.g., riders, motorcycles) hinder autonomous driving performance. To address this, we propose a programmatic weather image generation method that synergizes large language models (LLMs) with Stable Diffusion. Our approach innovatively integrates LLM-driven prompt engineering and tail-class-balanced sampling to jointly ensure weather diversity, scene plausibility, and high-fidelity long-tail class synthesis. Furthermore, we design a segmentation-model-agnostic domain generalization training framework compatible with any state-of-the-art (SOTA) segmenter. Evaluated on three cross-domain benchmarks (Cityscapes→ACDC, etc.), our method boosts the HRDA model’s mean Intersection-over-Union (mIoU) by 13.9%, significantly improving robustness to unseen weather conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this work, we propose a novel approach, namely WeatherDG, that can generate realistic, weather-diverse, and driving-screen images based on the cooperation of two foundation models, i.e, Stable Diffusion (SD) and Large Language Model (LLM). Specifically, we first fine-tune the SD with source data, aligning the content and layout of generated samples with real-world driving scenarios. Then, we propose a procedural prompt generation method based on LLM, which can enrich scenario descriptions and help SD automatically generate more diverse, detailed images. In addition, we introduce a balanced generation strategy, which encourages the SD to generate high-quality objects of tailed classes under various weather conditions, such as riders and motorcycles. This segmentation-model-agnostic method can improve the generalization ability of existing models by additionally adapting them with the generated synthetic data. Experiments on three challenging datasets show that our method can significantly improve the segmentation performance of different state-of-the-art models on target domains. Notably, in the setting of ''Cityscapes to ACDC'', our method improves the baseline HRDA by 13.9% in mIoU.

Problem

• Image Classification
• Weather Variability
• Rare Object Recognition

Innovation

• WeatherDG
• Stable Diffusion
• Large Language Model