HIMEC: Directional Change Representation and Fixed-Interface Decoding for Remote Sensing Image Change Captioning

2026-08-12

AI Summary

This work addresses limitations in existing remote sensing image change captioning methods, which directly fuse visual features into the decoder while neglecting explicit modeling of intermediate change structures and suffering from a mismatch between training and inference inputs. To overcome these issues, we propose the HIMEC framework, which introduces a directional change representation (DCR) to disentangle semantics corresponding to appearance, disappearance, and shared regions. A fixed-interface scene decoder ensures consistent inputs during both training and inference, while an auxiliary phrase decoder provides additional supervisory signals. By integrating multi-stream feature separation, a learnable query encoder, and a cascaded local-to-global conditioning mechanism, HIMEC significantly enhances change semantic modeling and system robustness. The method achieves CIDEr scores of 142.81±0.60 on LEVIR-CC and 75.67–76.99 on SECOND-CC, substantially outperforming current state-of-the-art approaches.

Abstract

Remote sensing image change captioning (RSICC) converts bitemporal imagery into a sentence describing semantic changes. Most RSICC methods condition caption decoders directly on fused visual features, leaving intermediate change structure and decoder-interface consistency less studied. We present HIMEC, combining Directional Change Representation (DCR) with fixed-interface decoding. DCR separates signed differences into appearance-oriented, disappearance-oriented, and shared-context streams before fusion. A learned-query encoder converts the fused representation into visually conditioned change-query tokens that form the scene decoder's only sample-dependent memory. A training-only auxiliary phrase decoder supplies caption-derived supervision. With a fixed zero input, the scene decoder maintains the same interface during training and inference. Separately, we evaluate a local-to-scene cascade conditioned on teacher-forced local states during training and autoregressive states at inference. On changed LEVIR-CC validation pairs, these states have a mean cosine distance of 0.69. Regime-matched conditioning recovers most of the associated deficit, whereas permuting state correspondence causes no detectable penalty. These findings are limited to the evaluated cascade. In a matched three-seed comparison, HIMEC reaches a Consensus-based Image Description Evaluation (CIDEr) score of $142.81\pm0.60$ on LEVIR-CC, versus $139.51\pm3.40$ for direct fused-feature memory. On SECOND-CC, fixed-zero and regime-matched diagnostic conditioning reach 75.67 and 76.99 CIDEr, respectively, versus 60.77 for the mismatched cascade. The source code will be made publicly available at https://github.com/ayshaashra/HIMEC upon publication.

Problem

• Remote sensing image change captioning
• Change representation
• Decoder-interface consistency
• Semantic change description
• Bitemporal imagery

Innovation

• Directional Change Representation
• Fixed-Interface Decoding
• Change Captioning
• Query-Based Memory
• Remote Sensing