🤖 AI Summary
This work addresses the challenges in training-free open-vocabulary segmentation—namely, the lack of a unified reasoning abstraction and the difficulty in simultaneously preserving spatial consistency and contextual awareness—by proposing a region-centric unified inference framework. Built upon frozen CLIP features, the method propagates visual information through a region-consistent interaction graph and integrates cross-window feature enhancement with offline reference memory retrieval to achieve end-to-end context restoration and retrieval alignment. For the first time, region-level abstraction is consistently applied throughout feature interaction, context modeling, and retrieval refinement. The approach substantially improves segmentation structural quality, inference robustness, and retrieval alignment across eight benchmarks, establishing a new state of the art in training-free open-vocabulary segmentation.
📝 Abstract
Training-free open-vocabulary segmentation remains limited by a missing inference abstraction. Frozen vision-language features are produced at patch level, yet dense prediction requires a unit that simultaneously governs feature interaction, spatial support, contextual recovery, and retrieval-based correction. We present SCI-CLIP, a segment-centric inference framework built around the principle that the same region abstraction should organize all stages of dense open-vocabulary prediction. SCI-CLIP first induces a region-consistent interaction graph over frozen visual tokens, then reconstructs dense features by propagating values over this graph, augmenting them with selective cross-window support only where local evidence is insufficient. The same segment abstraction is subsequently used to construct and query an offline reference memory, aligning exemplar retrieval with the units on which prediction is made. SCI-CLIP turns frozen CLIP-style features into spatially coherent, context-aware, and retrieval-compatible dense predictions without any training. SCI-CLIP consistently improves the structural quality of dense predictions, the robustness of contextual reasoning, and the alignment of exemplar-based correction, yielding stronger open-vocabulary segmentation across eight benchmarks. Project code is available at: https://github.com/mzamini92/SCICLIP.