Sink vs. diagonal patterns as mechanisms for attention switch and oversmoothing prevention

2026-05-08

AI Summary

This work investigates how attention mechanisms in Transformers mitigate oversmoothing through sink and diagonal patterns that enable attention switching. By integrating geometric analysis, theoretical proofs, and empirical validation, the study establishes—for the first time—an equivalence between sink tokens and hard attention switching, clarifies the precise conditions under which sinks effectively prevent oversmoothing, and quantitatively explains why pretrained models exhibit a preference for sink-based representations. The paper further introduces a diagonal pattern that permits self-communication as a more flexible mechanism for suppressing oversmoothing, thereby generalizing the applicability of attention switching. Additionally, it provides a quantitative comparison of the representational costs of sink versus diagonal patterns and elucidates the mechanistic conditions under which attention layers degenerate into MLP-like behavior.

Abstract

This paper studies the role of sinks and diagonal patterns as attention switch and anti-oversmoothing mechanisms. We analyze geometric conditions under which sinks can be represented, showing a necessary alignment between the embedding of the sink and all other embeddings. Next, we refine the current understanding of the role of sinks in oversmoothing prevention: we specify the conditions under which dense attention provably smooths more than sparse attention, and empirically verify that such conditions are often satisfied in practice. We further prove an equivalence between sinks and hard attention switch, in which the output of the attention is identically 0. Finally, we relax the hard attention switch by allowing token self-communication: we provide a quantitative comparison of the costs of representing sinks vs.\ diagonal patterns, showing why sinks are favored in pretrained transformers. The introduction and analysis of diagonal patterns and the generalization of the attention switch close the gap between what oversmoothing prevention requires and what sinks provide, while also establishing when and why attention layers act like MLPs if token communication is not necessary.

Problem

• attention switch
• oversmoothing prevention
• sinks
• diagonal patterns
• transformers

Innovation

• sink
• diagonal patterns
• attention switch
• oversmoothing prevention
• hard attention