SatBLIP: Context Understanding and Feature Identification from Satellite Imagery with Vision-Language Learning

2026-04-15

AI Summary

This study addresses the limitations of existing rural vulnerability assessments, which often rely on coarse-grained indicators that fail to capture localized risk factors such as housing quality, road accessibility, and land surface patterns. To overcome this, the authors propose the first vision-language learning framework tailored for satellite imagery, leveraging GPT-4o to generate structured image captions, fine-tuning a satellite-adapted BLIP model, and integrating embeddings from CLIP and large language models. By incorporating attention mechanisms and SHAP-based attribution, the approach enables semantic interpretation of rural scenes and accurate prediction of county-level Social Vulnerability Index (SVI). This method moves beyond conventional remote sensing practices that depend on handcrafted features or generic models, introducing a customized vision-language architecture for rural risk assessment that precisely identifies key drivers—including roof conditions, street width, and vegetation cover—thereby significantly enhancing both interpretability and robustness.

Technology Category

• Computer Vision: Language and Vision
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Rural environmental risks are shaped by place-based conditions (e.g., housing quality, road access, land-surface patterns), yet standard vulnerability indices are coarse and provide limited insight into risk contexts. We propose SatBLIP, a satellite-specific vision-language framework for rural context understanding and feature identification that predicts county-level Social Vulnerability Index (SVI). SatBLIP addresses limitations of prior remote sensing pipelines-handcrafted features, manual virtual audits, and natural-image-trained VLMs-by coupling contrastive image-text alignment with bootstrapped captioning tailored to satellite semantics. We use GPT-4o to generate structured descriptions of satellite tiles (roof type/condition, house size, yard attributes, greenery, and road context), then fine-tune a satellite-adapted BLIP model to generate captions for unseen images. Captions are encoded with CLIP and fused with LLM-derived embeddings via attention for SVI estimation under spatial aggregation. Using SHAP, we identify salient attributes (e.g., roof form/condition, street width, vegetation, cars/open space) that consistently drive robust predictions, enabling interpretable mapping of rural risk environments.

Problem

• rural vulnerability
• satellite imagery
• context understanding
• social vulnerability index
• environmental risk

Innovation

• vision-language learning
• satellite imagery
• social vulnerability index
• interpretable AI
• contrastive image-text alignment