A Novel Lightweight Transformer with Edge-Aware Fusion for Remote Sensing Image Captioning

2025-06-11

AI Summary

To address the high computational cost and neglect of fine-grained geometric structures—such as edges and contours—in Transformer-based remote sensing image captioning models, this paper proposes an edge-aware lightweight collaborative architecture. Our method introduces a lightweight Transformer encoder, an edge-detection-guided feature enhancement module to explicitly model boundary information, and a multi-stage knowledge distillation strategy to efficiently compress the GPT-2 decoder. This is the first work to jointly model semantic understanding and geometric structure perception in remote sensing captioning. Evaluated on multiple benchmark datasets, our approach achieves +3.2 BLEU-4 and +5.7 CIDEr improvements over prior state-of-the-art methods, while accelerating inference by 2.1× and reducing model parameters by 64%.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Machine Learning: Learning on the Edge & Model Compression
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Transformer-based models have achieved strong performance in remote sensing image captioning by capturing long-range dependencies and contextual information. However, their practical deployment is hindered by high computational costs, especially in multi-modal frameworks that employ separate transformer-based encoders and decoders. In addition, existing remote sensing image captioning models primarily focus on high-level semantic extraction while often overlooking fine-grained structural features such as edges, contours, and object boundaries. To address these challenges, a lightweight transformer architecture is proposed by reducing the dimensionality of the encoder layers and employing a distilled version of GPT-2 as the decoder. A knowledge distillation strategy is used to transfer knowledge from a more complex teacher model to improve the performance of the lightweight network. Furthermore, an edge-aware enhancement strategy is incorporated to enhance image representation and object boundary understanding, enabling the model to capture fine-grained spatial details in remote sensing images. Experimental results demonstrate that the proposed approach significantly improves caption quality compared to state-of-the-art methods.

Problem

• High computational costs hinder Transformer deployment in remote sensing captioning
• Existing models overlook fine-grained structural features like edges and boundaries
• Need lightweight architecture with edge-aware fusion for better spatial detail capture

Innovation

• Lightweight transformer with reduced encoder dimensionality
• Distilled GPT-2 decoder for efficient processing
• Edge-aware fusion enhances fine-grained spatial details