Evaluating Remote Sensing Image Captions Beyond Metric Biases

2026-04-22

AI Summary

This work addresses the limitations of existing reference-dependent evaluation metrics for remote sensing image captioning, which are prone to annotator-style biases that obscure the true capabilities of multimodal large language models (MLLMs) and mislead judgments about the necessity of fine-tuning. To overcome this, the authors propose ReconScore, a reference-free metric that assesses semantic fidelity by reconstructing the original visual content from generated captions. Building upon ReconScore, they introduce RemoteDescriber—a training-free method incorporating an iterative self-correction mechanism to enhance caption accuracy. Experiments demonstrate that off-the-shelf MLLMs outperform fine-tuned counterparts in genuine zero-shot settings, and RemoteDescriber achieves state-of-the-art performance across three remote sensing captioning benchmarks. Furthermore, ReconScore is shown to be more reliable and equitable than conventional evaluation metrics.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Security and Privacy: Large-scale security measurements

Abstract

The core objective of image captioning is to achieve lossless semantic compression from visual signals into textual modalities. However, the reliance on manually curated reference texts for evaluation essentially forces models to mimic specific human annotation styles, thereby masking the true descriptive capabilities of advanced foundation models. This systemic misalignment prompts a critical question: Is task-specific fine-tuning truly necessary for Remote Sensing Image Captioning, or is the perceived performance gap merely an artifact of flawed evaluation criteria? To investigate this discrepancy, we propose ReconScore, a novel reference-free evaluation metric. Rather than computing textual similarities, we assess caption quality by its capability to reconstruct the original visual elements solely from the generated text, effectively neutralizing human annotation biases. Applying this metric, we uncover a profound, counterintuitive truth: inherently powerful, unfine-tuned MLLMs surpass their fine-tuned counterparts in authentic zero-shot RSIC tasks. Driven by this structural discovery, we introduce RemoteDescriber, a completely training-free generation methodology. By employing ReconScore as a self-correction mechanism, we iteratively refine the semantic precision of MLLM outputs without any computational fine-tuning overhead. Comprehensive experiments demonstrate that RemoteDescriber achieves state-of-the-art performance on three datasets. Furthermore, we validate ReconScore's reliability and analyze the flaws of traditional metrics. Our code is available at https://github.com/hhu-czy/RemoteDescriber.

Problem

• Remote Sensing Image Captioning
• evaluation bias
• reference-free evaluation
• foundation models
• semantic compression

Innovation

• reference-free evaluation
• ReconScore
• RemoteDescriber
• zero-shot RSIC
• semantic reconstruction