🤖 AI Summary
Large Vision Transformers (ViTs) suffer from distorted attention maps, reduced interpretability, and suboptimal dense prediction performance due to an inherent trade-off between global and local feature modeling—specifically, the suppression of local information fusion by dominant global tokens. Method: Through systematic attention analysis, token attribution, and architectural diagnosis—empirically validated on models including DINOv2—we identify a fundamental structural equivalence between the CLS token and explicit register tokens. Contribution/Results: We establish, for the first time, that the CLS token functions as an *implicit register*, jointly governing global representation with explicit registers while impeding local feature integration. This challenges prevailing assumptions about attention interpretability in ViTs and provides a novel theoretical framework for understanding representational bottlenecks. Our findings inform principled design strategies for next-generation vision models that simultaneously achieve robust local–global modeling and enhanced interpretability.
📝 Abstract
Recent work has shown that the attention maps of the widely popular DINOv2 model exhibit artifacts, which hurt both model interpretability and performance on dense image tasks. These artifacts emerge due to the model repurposing patch tokens with redundant local information for the storage of global image information. To address this problem, additional register tokens have been incorporated in which the model can store such information instead. We carefully examine the influence of these register tokens on the relationship between global and local image features, showing that while register tokens yield cleaner attention maps, these maps do not accurately reflect the integration of local image information in large models. Instead, global information is dominated by information extracted from register tokens, leading to a disconnect between local and global features. Inspired by these findings, we show that the CLS token itself, which can be interpreted as a register, leads to a very similar phenomenon in models without explicit register tokens. Our work shows that care must be taken when interpreting attention maps of large ViTs. Further, by clearly attributing the faulty behaviour to register and CLS tokens, we show a path towards more interpretable vision models.