Assessing Building Heat Resilience Using UAV and Street-View Imagery with Coupled Global Context Vision Transformer

2026-01-16

AI Summary

This study addresses the heightened health risks from heat exposure in high-density cities of the Global South, where low-cost building materials and high-thermal-capacity surfaces exacerbate urban heat stress. To enable scalable assessment of building-level thermal resilience, the authors propose a dual-modality cross-view learning framework that, for the first time, integrates drone and street-level imagery with HotSat-1 thermal infrared data. By coupling these data streams through a Context-Guided Convolutional Vision Transformer (CGCViT), the model captures building attributes linked to thermal behavior. The approach outperforms the best single-modality baseline by 9.3% and enables household-level heat exposure mapping in Dar es Salaam. Results reveal the critical mitigating roles of vegetation cover, roof color, and material, as well as a spatial association between heat exposure and socioeconomic disadvantage, offering empirical support for equitable climate adaptation strategies.

Technology Category

• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multimodal Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Climate change is intensifying human heat exposure, particularly in densely built urban centers of the Global South. Low-cost construction materials and high thermal-mass surfaces further exacerbate this risk. Yet scalable methods for assessing such heat-relevant building attributes remain scarce. We propose a machine learning framework that fuses openly available unmanned aerial vehicle (UAV) and street-view (SV) imagery via a coupled global context vision transformer (CGCViT) to learn heat-relevant representations of urban structures. Thermal infrared (TIR) measurements from HotSat-1 are used to quantify the relationship between building attributes and heat-associated health risks. Our dual-modality cross-view learning approach outperforms the best single-modality models by up to $9.3\%$, demonstrating that UAV and SV imagery provide valuable complementary perspectives on urban structures. The presence of vegetation surrounding buildings (versus no vegetation), brighter roofing (versus darker roofing), and roofing made of concrete, clay, or wood (versus metal or tarpaulin) are all significantly associated with lower HotSat-1 TIR values. Deployed across the city of Dar es Salaam, Tanzania, the proposed framework illustrates how household-level inequalities in heat exposure - often linked to socio-economic disadvantage and reflected in building materials - can be identified and addressed using machine learning. Our results point to the critical role of localized, data-driven risk assessment in shaping climate adaptation strategies that deliver equitable outcomes.

Problem

• heat resilience
• urban heat exposure
• building attributes
• climate adaptation
• thermal risk assessment

Innovation

• Coupled Global Context Vision Transformer
• UAV and Street-View Fusion
• Heat Resilience Assessment
• Thermal Infrared Remote Sensing
• Urban Climate Equity