🤖 AI Summary
This work addresses the issue of semantic diffusion in local representations of Vision Transformers under prolonged training, which degrades performance in dense prediction tasks. To mitigate this over-smoothing effect while preserving global context, the authors propose replacing the standard softmax attention with entmax-1.5 sparse attention, thereby enhancing semantic focus on local tokens. Implemented within the DINOv1 ViT-S/16 architecture, the method maintains the complementary relationship between the [CLS] token and local features. Without compromising linear probing accuracy on ImageNet-1K, it achieves substantial gains in segmentation performance, yielding mIoU scores of 48.78 on PASCAL VOC, 21.97 on ADE20K, and 37.87 on Cityscapes.
📝 Abstract
Vision Transformers (ViTs) can learn strong image-level representations while their patch representations become less effective for dense prediction during prolonged training. We revisit this dense degradation phenomenon and argue that it is not fully explained by high-norm artifacts alone. Instead, we characterize \emph{semantic diffusion}: an optimization shortcut in which global semantic information spreads through patch tokens beyond what is locally justified. Our analysis shows that dense representation quality is not captured by locality alone: shallow features can remain better aligned with foreground regions yet underperform deeper features, and \texttt{[CLS]} features remain complementary for dense prediction. These observations suggest that the goal should not be to remove global context, but to make token interactions more selective. We therefore study sparse attention as a minimal intervention, replacing softmax attention with entmax-1.5 while preserving global token connectivity. On DINOv1 ViT-S/16 trained for 200 epochs on ImageNet-1K, this change preserves ImageNet linear probing accuracy and substantially improves semantic segmentation performance: VOC mIoU increases from 42.80 to 48.78, ADE20K from 19.85 to 21.97, and Cityscapes from 36.79 to 37.87. These results suggest that selective token mixing is a simple and effective bias for improving dense ViT representations.