Target Refocusing via Attention Redistribution for Open-Vocabulary Semantic Segmentation: An Explainability Perspective

2025-11-20

AI Summary

In open-vocabulary semantic segmentation (OVSS), CLIP-based models suffer from coarse-grained pixel-level vision-language alignment and attention dispersion to irrelevant regions. Method: This paper introduces, for the first time, an interpretability-driven analysis revealing attention degradation in dense prediction tasks, and proposes a training-free attention refocusing mechanism: dimension-specific filtering identifies target-relevant attention channels, followed by weight redistribution to amplify responses in target regions—mimicking human attention recapture. Crucially, it achieves fine-grained pixel-level alignment without CLIP fine-tuning. Contribution/Results: The method attains state-of-the-art performance on eight major benchmarks, demonstrating high inference efficiency and strong generalization. It validates that interpretability-guided design is both effective and feasible for enhancing dense prediction capabilities of foundational vision-language models.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Open-vocabulary semantic segmentation (OVSS) employs pixel-level vision-language alignment to associate category-related prompts with corresponding pixels. A key challenge is enhancing the multimodal dense prediction capability, specifically this pixel-level multimodal alignment. Although existing methods achieve promising results by leveraging CLIP's vision-language alignment, they rarely investigate the performance boundaries of CLIP for dense prediction from an interpretability mechanisms perspective. In this work, we systematically investigate CLIP's internal mechanisms and identify a critical phenomenon: analogous to human distraction, CLIP diverts significant attention resources from target regions to irrelevant tokens. Our analysis reveals that these tokens arise from dimension-specific over-activation; filtering them enhances CLIP's dense prediction performance. Consequently, we propose ReFocusing CLIP (RF-CLIP), a training-free approach that emulates human distraction-refocusing behavior to redirect attention from distraction tokens back to target regions, thereby refining CLIP's multimodal alignment granularity. Our method achieves SOTA performance on eight benchmarks while maintaining high inference efficiency.

Problem

• Enhancing pixel-level multimodal alignment for open-vocabulary segmentation
• Addressing CLIP's attention diversion to irrelevant tokens
• Improving dense prediction performance without additional training

Innovation

• Redistributes attention from irrelevant tokens to targets
• Filters dimension-specific over-activation in CLIP
• Training-free approach mimicking human refocusing behavior