Object detection in adverse weather conditions for autonomous vehicles using Instruct Pix2Pix

2025-05-13

AI Summary

Adverse weather conditions—such as rain, fog, and snow—severely degrade the performance of autonomous driving object detectors. To address this, this work quantifies, for the first time, the impact of weather degradation on the mean Average Precision (mAP) of mainstream detectors (YOLOv10 and Faster R-CNN). We propose an instruction-driven, controllable weather augmentation paradigm based on Instruct Pix2Pix, leveraging diffusion models to generate high-fidelity, semantically consistent weather-corrupted images for training data expansion. Evaluated on the real-world BDD100K and ACDC datasets, our method improves the average mAP of YOLOv10 and Faster R-CNN by up to 12.3%. It further demonstrates strong generalization across both CARLA simulations and real-world scenarios. This work establishes a novel, interpretable, and controllable data augmentation framework for robust perception under adverse weather conditions.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Enhancing the robustness of object detection systems under adverse weather conditions is crucial for the advancement of autonomous driving technology. This study presents a novel approach leveraging the diffusion model Instruct Pix2Pix to develop prompting methodologies that generate realistic datasets with weather-based augmentations aiming to mitigate the impact of adverse weather on the perception capabilities of state-of-the-art object detection models, including Faster R-CNN and YOLOv10. Experiments were conducted in two environments, in the CARLA simulator where an initial evaluation of the proposed data augmentation was provided, and then on the real-world image data sets BDD100K and ACDC demonstrating the effectiveness of the approach in real environments. The key contributions of this work are twofold: (1) identifying and quantifying the performance gap in object detection models under challenging weather conditions, and (2) demonstrating how tailored data augmentation strategies can significantly enhance the robustness of these models. This research establishes a solid foundation for improving the reliability of perception systems in demanding environmental scenarios, and provides a pathway for future advancements in autonomous driving.

Problem

• Enhancing object detection robustness in adverse weather for autonomous vehicles
• Using Instruct Pix2Pix to generate weather-augmented datasets for model training
• Improving Faster R-CNN and YOLOv10 performance in real-world weather conditions

Innovation

• Uses Instruct Pix2Pix for weather-based data augmentation
• Evaluates in CARLA simulator and real-world datasets
• Enhances Faster R-CNN and YOLOv10 robustness