Open-SAT: LLM-Guided Query Embedding Refinement for Open-Vocabulary Object Retrieval in Satellite Imagery

2026-05-06

AI Summary

This work addresses the challenges of open-vocabulary natural language querying in satellite image retrieval, particularly the difficulties of handling an open set of categories and achieving effective visual-semantic alignment. The authors propose a training-free query embedding refinement method that leverages large language models (LLMs) during inference to dynamically refine user queries, thereby enhancing their semantic alignment with satellite image content. By integrating CLIP-derived image embeddings with a vector database, the approach enables efficient retrieval without requiring additional training or threshold tuning. Notably, this is the first study to employ LLMs for context-aware query optimization in open-vocabulary satellite image retrieval. Experiments demonstrate significant performance gains, with F1 scores improving by up to 16.04% across three public benchmarks while maintaining comparable retrieval scalability to existing methods.

Abstract

In satellite applications, user queries often take the form of open-ended natural language, extending beyond a fixed set of predefined categories. This open-vocabulary nature poses significant challenges for retrieving relevant image tiles, as the retrieval system must generalize to a wide range of unseen objects and concepts. While vision-language models (VLMs) such as CLIP are widely used for text-image retrieval, even fine-tuned variants often struggle to accurately align such queries with satellite imagery. To address this, we propose Open-SAT, a training-free query embedding refinement algorithm that operates at inference time to improve alignment between user queries and satellite image content. Open-SAT uses VLMs to compute embeddings for image tiles, which are stored in a vector database for efficient retrieval. At query time, it leverages Large Language Models (LLMs) to refine the text embeddings by incorporating contextual information about objects of interest and their surroundings. A threshold-free retrieval mechanism further enhances accuracy and efficiency. Experimental results in three public benchmarks demonstrate that Open-SAT improves the F1 score by up to 16.04%, while retrieving a comparable number of image tiles. These results demonstrate the effectiveness of Open-SAT in open-vocabulary satellite image retrieval, leveraging LLM guidance without the need for additional training or supervision.

Problem

• open-vocabulary object retrieval
• satellite imagery
• query embedding
• vision-language models
• natural language queries

Innovation

• open-vocabulary retrieval
• LLM-guided refinement
• satellite imagery
• query embedding
• training-free adaptation