Repurposing CLIP to Localize at Pixel Level

2026-07-06

AI Summary

This work addresses the challenge of effectively extending the open-vocabulary, image-level recognition capabilities of vision-language models like CLIP to pixel-level dense prediction tasks. To this end, the authors propose CLIPix, a novel framework that achieves high-resolution, arbitrary-category segmentation in open-vocabulary settings—marking the first such application of CLIP for this purpose. The method leverages attention mechanisms from CLIP’s classification process to extract object-related regions as pixel-level cues, and enhances detail accuracy through a noise-resistant refinement strategy and a localization embedding fusion scheme, all while preserving strong semantic generalization. Experimental results demonstrate state-of-the-art performance on both PASCAL and COCO benchmarks, significantly advancing the practicality and accuracy of open-vocabulary segmentation.

Abstract

Large-scale Vision-Language Models like CLIP have demonstrated impressive open-set localization capabilities at the image level. However, adapting this capability to pixel-level dense prediction poses challenges due to global feature biases. In this paper, we introduce CLIPix, a simple yet effective framework that repurposes CLIP to perform pixel-level localization. By tracing back CLIP's classification process, CLIPix identifies object-specific attentive regions and repurposes them as pixel-level localization cues. To address noise introduced by global biases, we propose a Noise-Resistant Correction strategy, refining these cues for more precise segmentation. Additionally, we introduce a Localization Embedding strategy to integrate both localization and enriched detail information, enabling accurate, high-resolution segmentation. Our approach preserves CLIP's generalization strength and unlocks its potential for segmenting arbitrary objects. Extensive experiments on the PASCAL and COCO datasets demonstrate that CLIPix achieves state-of-the-art performance, underscoring its effectiveness.

Problem

• Vision-Language Models
• Pixel-level Localization
• CLIP
• Dense Prediction
• Open-set Segmentation

Innovation

• CLIPix
• pixel-level localization
• Vision-Language Model
• Noise-Resistant Correction
• Localization Embedding