Open-Vocabulary Semantic Segmentation in Remote Sensing via Hierarchical Attention Masking and Model Composition

2026-02-27

AI Summary

This work addresses the performance degradation of vision-language models such as CLIP in open-vocabulary semantic segmentation of remote sensing imagery, which stems from improper interactions within self-attention layers. To this end, the authors propose ReSeg-CLIP, a training-free approach that introduces hierarchical attention masking and a multi-model fusion mechanism tailored for remote sensing scenarios. Specifically, multi-scale masks generated by SAM are leveraged to constrain CLIP’s self-attention interactions, while multiple remote sensing–specialized CLIP variants are fused through a weighted ensemble guided by text-prompt-based representation quality assessment. Evaluated under a zero-shot setting on three remote sensing benchmark datasets, ReSeg-CLIP achieves state-of-the-art performance, significantly enhancing CLIP’s open-vocabulary comprehension capabilities in remote sensing contexts.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

In this paper, we propose ReSeg-CLIP, a new training-free Open-Vocabulary Semantic Segmentation method for remote sensing data. To compensate for the problems of vision language models, such as CLIP in semantic segmentation caused by inappropriate interactions within the self-attention layers, we introduce a hierarchical scheme utilizing masks generated by SAM to constrain the interactions at multiple scales. We also present a model composition approach that averages the parameters of multiple RS-specific CLIP variants, taking advantage of a new weighting scheme that evaluates representational quality using varying text prompts. Our method achieves state-of-the-art results across three RS benchmarks without additional training.

Problem

• Open-Vocabulary Semantic Segmentation
• Remote Sensing
• Vision Language Models
• Self-Attention
• Semantic Segmentation

Innovation

• Open-Vocabulary Semantic Segmentation
• Hierarchical Attention Masking
• Model Composition
• Remote Sensing
• CLIP