Efficient Remote Sensing Instance Segmentation with Linear-Time State Space Distilled Visual Foundation Models

2026-06-23

AI Summary

This work addresses the challenge of efficiently applying vision foundation models to dense prediction tasks in remote sensing image instance segmentation, where the quadratic computational complexity of self-attention mechanisms hinders scalability. To overcome this limitation, the authors propose RS4D, a novel approach that, for the first time, integrates state space models (SSMs) with adaptive noise-mask knowledge distillation to enable efficient knowledge transfer from large vision transformers (ViTs) to lightweight SSM-based architectures with linear time complexity. The method incorporates a compact encoder and a multi-head segmentation design, achieving an 8× reduction in parameters and a 9× decrease in FLOPs on the SSDD, WHU, and NWPU datasets while maintaining or surpassing the segmentation accuracy of current ViT- and CNN-based methods.

Abstract

The computational complexity of Transformers scales quadratically with the number of tokens, which significantly constrains the efficiency of vision models, particularly recent ViT-based foundation models in dense prediction tasks. Instance segmentation, a typical dense visual prediction task in the remote sensing field, faces similar challenges. In this paper, inspired by the recent advances of knowledge distillation in large language models, we introduce RS4D - a new remote sensing instance segmentation method with linear computational complexity, which addresses the inefficiency of long sequence modeling through distilled state space modeling (SSM). We propose an adaptive noise and masking knowledge distillation training method for pre-training lightweight SSM backbones, which effectively compresses knowledge from the vast self-attention space into a compact, dense linear state space. We also design a remote sensing image instance segmentation architecture based on this lightweight visual encoder, where we explore variants of three different backbones and two segmentation heads. Extensive experiments are conducted on multiple benchmark datasets, including SSDD, WHU, and NWPU. Compared to ViT-based approaches, our proposed SSM backbone achieves an 8x reduction in parameters and a 9x reduction in FLOPs while maintaining comparable or superior accuracy to both ViT- and CNN-based instance segmentation methods. The implementation codes have been publicly available at https://github.com/QinzheYang/RS4D.

Problem

• remote sensing
• instance segmentation
• computational efficiency
• dense prediction
• vision foundation models

Innovation

• State Space Model
• Knowledge Distillation
• Linear Complexity
• Remote Sensing Instance Segmentation
• Efficient Vision Model