TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data

2024-10-08

arXiv.org

AI Summary

Current vision-language assistants (VLAs) support only single-frame remote sensing image understanding, limiting their ability to model dynamic Earth surface changes and hindering deployment in real-world spatiotemporal analysis tasks. To address this, we propose the first VLA framework tailored for conversational reasoning over time-series remote sensing imagery, overcoming the single-frame constraint and unifying support for diverse spatiotemporal reasoning tasks—including building change detection, semantic evolution identification, and temporal scene classification. Methodologically, we introduce a novel spatiotemporal instruction-tuning paradigm, construct a dedicated instruction dataset covering both single-image and time-series tasks, and integrate spatiotemporal feature alignment with multi-granularity prompt engineering. Experiments demonstrate that our model significantly outperforms general-purpose VLAs (e.g., GPT-4o, Gemini 1.5 Pro) across multiple spatiotemporal reasoning benchmarks, achieves strong zero-shot cross-task generalization, and matches the performance of task-specific models. The code, models, and dataset are fully open-sourced.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large vision and language assistants have enabled new capabilities for interpreting natural images. These approaches have recently been adapted to earth observation data, but they are only able to handle single image inputs, limiting their use for many real-world tasks. In this work, we develop a new vision and language assistant called TEOChat that can engage in conversations about temporal sequences of earth observation data. To train TEOChat, we curate an instruction-following dataset composed of many single image and temporal tasks including building change and damage assessment, semantic change detection, and temporal scene classification. We show that TEOChat can perform a wide variety of spatial and temporal reasoning tasks, substantially outperforming previous vision and language assistants, and even achieving comparable or better performance than several specialist models trained to perform specific tasks. Furthermore, TEOChat achieves impressive zero-shot performance on a change detection and change question answering dataset, outperforms GPT-4o and Gemini 1.5 Pro on multiple temporal tasks, and exhibits stronger single image capabilities than a comparable single image instruction-following model on scene classification, visual question answering, and captioning. We publicly release our data, model, and code at https://github.com/ermongroup/TEOChat .

Problem

• Visual-Linguistic Assistants
• Earth Observation Data
• Temporal Information Analysis

Innovation

• TEOChat
• spatiotemporal reasoning
• Earth observation images