DamageArbiter: A CLIP-Enhanced Multimodal Arbitration Framework for Hurricane Damage Assessment from Street-View Imagery

2026-03-16

AI Summary

This study addresses the limited interpretability and reliability of existing street-level hurricane damage assessment methods, which often suffer from ambiguous visual cues and overconfident misclassifications. To overcome these limitations, the authors propose DamageArbiter, a CLIP-based multimodal arbitration framework that reconciles prediction discrepancies between unimodal vision models (e.g., ViT-B/32) and multimodal image-text models via a lightweight logistic regression meta-classifier. The framework further integrates human-annotated and large language model–generated textual descriptions to inform its arbitration decisions. Evaluated on a dataset of 2,556 post-disaster street-view images, DamageArbiter achieves an accuracy of 82.79%, surpassing the strongest baseline by 8.46 percentage points, while substantially reducing misclassifications and enhancing both model interpretability and robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Analyzing street-view imagery with computer vision models for rapid, hyperlocal damage assessment is becoming popular and valuable in emergency response and recovery, but traditional models often act like black boxes, lacking interpretability and reliability. This study proposes a multimodal disagreement-driven Arbitration framework powered by Contrastive Language-Image Pre-training (CLIP) models, DamageArbiter, to improve the accuracy, interpretability, and robustness of damage estimation from street-view imagery. DamageArbiter leverages the complementary strengths of unimodal and multimodal models, employing a lightweight logistic regression meta-classifier to arbitrate cases of disagreement. Using 2,556 post-disaster street-view images, paired with both manually generated and large language model (LLM)-generated text descriptions, we systematically compared the performance of unimodal models (including image-only and text-only models), multimodal CLIP-based models, and DamageArbiter. Notably, DamageArbiter improved the accuracy from 74.33% (ViT-B/32, image-only) to 82.79%, surpassing the 80% accuracy threshold and achieving an absolute improvement of 8.46% compared to the strongest baseline model. Beyond improvements in overall accuracy, compared to visual models relying solely on images, DamageArbiter, through arbitration of discrepancies between unimodal and multimodal predictions, mitigates common overconfidence errors in visual models, especially in situations where disaster visual cues are ambiguous or subject to interference, reducing overconfidence but incorrect predictions. We further mapped and analyzed geo-referenced predictions and misclassifications to compare model performance across locations. Overall, this work advances street-view-based disaster assessment from coarse severity classification toward a more reliable and interpretable framework.

Problem

• hurricane damage assessment
• street-view imagery
• model interpretability
• overconfidence
• multimodal disagreement

Innovation

• multimodal arbitration
• CLIP
• disaster damage assessment
• interpretability
• street-view imagery