Sparse Attention for Dense Open-Vocabulary Prediction in CLIP

2026-07-08

AI Summary

This work addresses the limitation of CLIP in dense open-vocabulary prediction tasks, where dense softmax attention disperses focus onto semantically irrelevant regions, introducing noise and weakening fine-grained spatial cues. The authors propose the first integration of the α-entmax transformation into CLIP’s visual self-attention mechanism, enabling data-dependent, training-free attention sparsification during inference by automatically masking out irrelevant tokens via an adaptive threshold. This approach effectively suppresses noise and enhances responses in salient regions, yielding significant performance gains on semantic segmentation benchmarks (Pascal VOC, Pascal Context, ADE20K) and fine-grained open-vocabulary detection (FG-OVD). Notably, the improvement correlates positively with the baseline attention’s degree of dispersion, highlighting the method’s efficacy in refining attention focus.

Abstract

Contrastive Language-Image Pre-training (CLIP) relies on softmax-based self-attention, a strictly positive distribution that assigns probability mass to every pair of tokens-even semantically irrelevant ones. While these dense softmax weights are effective for gathering broad context during pre-training, they spread attention across many low-salience tokens, producing noise that obscures the fine-grained, spatially localized cues required for dense, open-vocabulary prediction. We study an inference-time substitution of the row-wise softmax in the final visual self-attention layers with the $α$-entmax transform, applied across both the standard query-key attention and self-correlation variants. Because entmax applies a data-dependent threshold that maps low scores exactly to zero, it acts as an implicit denoiser, zeroing contextually irrelevant dependencies while redistributing mass onto the most relevant tokens. We evaluate on open-vocabulary tasks-dense semantic segmentation (Pascal VOC, Pascal Context, ADE20K) and fine-grained retrieval (FG-OVD)-and find the gain from attention sparsification is proportional to how much the baseline attention spreads off the target class.

Problem

• Sparse Attention
• Dense Prediction
• Open-Vocabulary
• CLIP
• Self-Attention

Innovation

• sparse attention
• α-entmax
• CLIP
• open-vocabulary prediction
• dense semantic segmentation