Where Does Texture Evidence Live in SAM? Features, Proposal Masks, and Texture Segmentation

2026-06-07

AI Summary

This work addresses the suboptimal performance of the foundational segmentation model SAM on texture-based segmentation tasks, where regions are defined by material properties or repetitive patterns. We systematically disentangle the multiple causes underlying SAM’s failures in such scenarios, demonstrating that its limitations stem not from “texture blindness” but from mismatches among representation, proposal support, readout mechanisms, and decision commitment. Building upon a frozen SAM backbone, we propose two readout strategies that require neither fine-tuning the backbone nor retraining the proposal generator: multi-scale feature clustering and proposal-bank-based supervised integration. Experiments on RWTD, STLD, an ADE20K subset, and ControlNet-synthesized data show that coarse-grained features preserve texture structure and that the proposal bank often contains texture-aligned masks—enabling direct selection of complete proposals in synthetic scenes, while natural scenes necessitate fragment assembly.

Abstract

Texture segmentation stresses foundation segmentation because meaningful regions are defined by material or repeated appearance rather than object identity. Segment Anything Models (SAMs) often fail by default on such texture-defined partitions, but this failure is ambiguous: the texture evidence may be absent, missing from the proposal bank, or present but selected or assembled incorrectly by an object-centric readout. We ask what texture-relevant evidence is already preserved in frozen SAM before adaptation. We study two frozen evidence spaces: multiscale features, probed with a minimal clustering readout, and the automatic proposal bank, treated as evidence for a supervised consolidation readout. SAM is frozen throughout; we do not fine-tune the backbone or retrain the proposal generator. Across RWTD, STLD, an ADE20K-selected refined-crop complement, and a ControlNet-stitched PTD bridge archive, frozen SAM is not a texture segmenter by default, but its failures are not simple texture blindness. Coarse frozen features preserve texture organization, and proposal banks often contain texture-aligned masks or fragments. Natural scenes more often require assembly and commitment over fragments, while cleaner synthetic cases more often reduce to selecting an already coherent proposal. Default mask failure should therefore be decomposed into representation evidence, proposal-bank support, readout mismatch, and commitment failure.

Problem

• texture segmentation
• Segment Anything Model
• feature representation
• proposal mask
• readout mechanism

Innovation

• texture segmentation
• Segment Anything Model
• frozen features
• proposal masks
• readout mechanism