Multimodal and Multiscale Spatial-Temporal Semantic Search and Recommendation with AI Foundation Models

2026-06-15

AI Summary

This study addresses the challenge of semantic search and recommendation for multimodal environmental event documents—such as anomalous ecological incident reports—in geographic information retrieval. The authors propose CAMERA, a multimodal event retrieval algorithm, and ASTRA, an adaptive spatiotemporal reranking algorithm. By integrating large language models (LLMs) and vision-language models (VLMs) to generate enhanced embeddings, the approach jointly optimizes scale-aware spatiotemporal relevance and semantic similarity for the first time, enabling multidimensional intelligent retrieval. Experiments on a local environmental observer network dataset demonstrate that the proposed VLM-enhanced method significantly outperforms text-only LLM baselines, substantially improving similarity ranking performance and facilitating deeper public and managerial understanding of localized environmental impacts.

Abstract

Semantic search and recommendation of similar documents, such as news and reports about unusual environmental events (e.g., a dead whale washed ashore in Alaska) that contain spatial and temporal information, is a critical task in Geographic Information Retrieval (GIR). This work presents a novel framework that leverages AI foundation models, including Large Language Models (LLMs) and Vision-Language Models (VLMs), to enable effective similarity search and ranking for such event documents. To support this goal, we introduce two new strategies: (1) CAMERA (Context-Aware Multimodal Event Retrieval Algorithm), which fuses textual and visual information to generate richer embeddings than those derived from text alone; and (2) ASTRA (Adaptive Spatial and Temporal Re-ranking Algorithm), which improves similarity ranking by incorporating scale-dependent spatiotemporal relevance alongside semantic similarity. Experimental results, using a dataset from the Local Environmental Observer Network, demonstrate that our VLM-enhanced methods outperform unimodal, LLM-based approaches in similarity ranking effectiveness. By automatically linking relevant event reports, the proposed framework helps both data curators and the general public gain deeper insights into environmental change and its localized impacts. These findings highlight the potential of AI foundation models to advance GIR through multifaceted, intelligent analysis that integrates key geographic concepts: space, time, scale, and semantics.

Problem

• Geographic Information Retrieval
• Semantic Search
• Multimodal
• Spatiotemporal
• Environmental Events

Innovation

• multimodal retrieval
• foundation models
• spatiotemporal relevance
• semantic search
• geographic information retrieval