FE-LWS: Refined Image-Text Representations via Decoder Stacking and Fused Encodings for Remote Sensing Image Captioning

2025-02-13

AI Summary

To address low caption accuracy in remote sensing image captioning caused by inadequate visual representation, this paper proposes a dual-stream CNN encoder coupled with a weighted stacked GRU decoder. We introduce a feature-level dual-encoder fusion mechanism and a layer-wise weighted averaging strategy for GRUs to enhance visual-semantic alignment. Furthermore, we incorporate contrastive-driven beam search during decoding, explicitly modeling semantic discrepancies between candidate captions and reference descriptions to improve both relevance and diversity. Evaluated on multiple remote sensing image captioning benchmarks, our method significantly outperforms Transformer- and LSTM-based baselines: BLEU-4 improves by over 4.2%, and CIDEr by up to 6.8%. These results demonstrate the effectiveness and state-of-the-art performance of our approach for domain-specific vision-language generation tasks.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Remote sensing image captioning aims to generate descriptive text from remote sensing images, typically employing an encoder-decoder framework. In this setup, a convolutional neural network (CNN) extracts feature representations from the input image, which then guide the decoder in a sequence-to-sequence caption generation process. Although much research has focused on refining the decoder, the quality of image representations from the encoder remains crucial for accurate captioning. This paper introduces a novel approach that integrates features from two distinct CNN based encoders, capturing complementary information to enhance caption generation. Additionally, we propose a weighted averaging technique to combine the outputs of all GRUs in the stacked decoder. Furthermore, a comparison-based beam search strategy is incorporated to refine caption selection. The results demonstrate that our fusion-based approach, along with the enhanced stacked decoder, significantly outperforms both the transformer-based state-of-the-art model and other LSTM-based baselines.

Problem

• Enhancing remote sensing image captioning accuracy
• Integrating dual CNN encoders for richer image features
• Improving decoder with weighted GRU outputs and beam search

Innovation

• Fused CNN encoders
• Weighted GRU averaging
• Comparison-based beam search