Good Representation, Better Explanation: Role of Convolutional Neural Networks in Transformer-Based Remote Sensing Image Captioning

2025-02-22

AI Summary

This paper addresses the suboptimal visual encoder design in transformer-based remote sensing image captioning (RSIC). To systematically evaluate encoder architectures, it empirically benchmarks twelve CNN backbones—including ResNet, EfficientNet, and ResNeXt—as visual encoders for RSIC. A novel two-stage evaluation paradigm—combining numerical clustering with human assessment—is proposed and applied for the first time to enable rigorous, multi-encoder comparison in RSIC. Experimental results demonstrate that ResNeXt-101 substantially improves caption quality: CIDEr increases by 12.7%, and human evaluation confirms enhanced descriptive accuracy and richer geospatial semantics. The study reveals that a powerful CNN encoder effectively compensates for transformers’ limitations in low-level texture modeling. It establishes a reproducible framework for encoder selection and evaluation in RSIC, providing both methodological guidance and empirical evidence for future model design.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Remote Sensing Image Captioning (RSIC) is the process of generating meaningful descriptions from remote sensing images. Recently, it has gained significant attention, with encoder-decoder models serving as the backbone for generating meaningful captions. The encoder extracts essential visual features from the input image, transforming them into a compact representation, while the decoder utilizes this representation to generate coherent textual descriptions. Recently, transformer-based models have gained significant popularity due to their ability to capture long-range dependencies and contextual information. The decoder has been well explored for text generation, whereas the encoder remains relatively unexplored. However, optimizing the encoder is crucial as it directly influences the richness of extracted features, which in turn affects the quality of generated captions. To address this gap, we systematically evaluate twelve different convolutional neural network (CNN) architectures within a transformer-based encoder framework to assess their effectiveness in RSIC. The evaluation consists of two stages: first, a numerical analysis categorizes CNNs into different clusters, based on their performance. The best performing CNNs are then subjected to human evaluation from a human-centric perspective by a human annotator. Additionally, we analyze the impact of different search strategies, namely greedy search and beam search, to ensure the best caption. The results highlight the critical role of encoder selection in improving captioning performance, demonstrating that specific CNN architectures significantly enhance the quality of generated descriptions for remote sensing images. By providing a detailed comparison of multiple encoders, this study offers valuable insights to guide advances in transformer-based image captioning models.

Problem

• Optimize encoder in transformer-based RSIC
• Evaluate CNN architectures for feature extraction
• Assess search strategies for caption quality

Innovation

• CNN architectures transformer encoder
• Human-centric evaluation caption quality
• Greedy and beam search strategies