Sub-Semantic Image Segmentation

2026-06-07

AI Summary

This work addresses a critical gap in existing image segmentation methods, which typically operate either at the texture or semantic level and lack an intermediate-granularity representation capable of capturing visually consistent regions describable in natural language. To bridge this gap, we introduce a novel paradigm—sub-semantic image segmentation—that aims to partition images into regions exhibiting both appearance consistency and linguistic describability. We propose DETECTURE, a framework that integrates vision-language models with the SAM 3 segmentation backbone, effectively mitigating three key failure modes: language leakage, prompt competition, and semantic distortion. Furthermore, we construct TextureADE, the first sub-semantic segmentation dataset, automatically derived from ADE20K. Extensive experiments demonstrate that our approach significantly outperforms existing methods across multiple benchmarks, validating both the efficacy of sub-semantic segmentation and the superiority of the DETECTURE framework.

Abstract

Images can be segmented based on visual cues (i.e., texture segmentation) or into objects (i.e., semantic segmentation). We propose a new category of sub-semantic image segmentation that blurs the line between the two. In sub-semantic image segmentation, language is not used to name whole objects. Instead, it is used to partition an image into stable appearance patterns that can be described by language. To do that, we couple a general-purpose vision-language model to SAM 3, a promptable segmentation backbone whose native text pathway can ground rich descriptions into masks. Simple coupling fails for a number of reasons that we identify in the paper, and we overcome them by introducing DETECTURE that resolves three concrete failure modes -- language leakage between texture regions, prompt competition inside the segmentation backbone, and semantic distortion at the language-to-mask interface. Since there is no dataset of sub-semantic image segmentation, we introduce one, termed TextureADE. The new dataset is derived from the ADE20K dataset using a system we designed. We compare DETECTURE to a number of baselines and find that it achieves the strongest performance on several datasets using different metrics. Code is available at https://github.com/Scientific-Computing-Lab/TextureDetecture.

Problem

• sub-semantic segmentation
• image segmentation
• vision-language model
• texture segmentation
• semantic segmentation

Innovation

• sub-semantic segmentation
• vision-language model
• DETECTURE
• TextureADE
• promptable segmentation