Post-Hurricane Debris Segmentation Using Fine-Tuned Foundational Vision Models

2025-04-17

AI Summary

Existing debris segmentation methods for post-hurricane damage assessment suffer from poor cross-event generalization, scarcity of annotated data, and difficulty achieving zero false positives in operational deployment. Method: We propose the first event-agnostic, robust debris segmentation framework, fine-tuning the CLIPSeg vision foundation model using only single-modality RGB aerial imagery; we further introduce a multi-annotator label fusion strategy and visual prompting engineering to enhance segmentation consistency and robustness under few-shot conditions. Contributions/Results: (1) First zero-false-positive debris segmentation across distinct hurricane events (Ian, Ida, Ike); (2) Release of a novel, finely annotated dataset comprising 1,200 aerial images from multiple hurricanes; (3) Achieves a Dice score of 0.70 on the unseen Hurricane Ida test set with zero false positives in non-debris regions. The lightweight framework enables real-time deployment for large-scale rapid post-disaster damage assessment.

Technology Category

• Computer Vision: Segmentation
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Timely and accurate detection of hurricane debris is critical for effective disaster response and community resilience. While post-disaster aerial imagery is readily available, robust debris segmentation solutions applicable across multiple disaster regions remain limited. Developing a generalized solution is challenging due to varying environmental and imaging conditions that alter debris' visual signatures across different regions, further compounded by the scarcity of training data. This study addresses these challenges by fine-tuning pre-trained foundational vision models, achieving robust performance with a relatively small, high-quality dataset. Specifically, this work introduces an open-source dataset comprising approximately 1,200 manually annotated aerial RGB images from Hurricanes Ian, Ida, and Ike. To mitigate human biases and enhance data quality, labels from multiple annotators are strategically aggregated and visual prompt engineering is employed. The resulting fine-tuned model, named fCLIPSeg, achieves a Dice score of 0.70 on data from Hurricane Ida -- a disaster event entirely excluded during training -- with virtually no false positives in debris-free areas. This work presents the first event-agnostic debris segmentation model requiring only standard RGB imagery during deployment, making it well-suited for rapid, large-scale post-disaster impact assessments and recovery planning.

Problem

• Detect hurricane debris accurately for disaster response
• Generalize debris segmentation across varying environmental conditions
• Overcome limited training data with fine-tuned models

Innovation

• Fine-tuned foundational vision models for debris segmentation
• Open-source dataset with 1,200 annotated aerial images
• Event-agnostic model using standard RGB imagery