Inhibited Self-Attention: Sharpening Focus in Vision Transformers

2026-07-14

AI Summary

Vision Transformers are prone to background distractions and often rely on spurious correlations for prediction. To address this, this work proposes Inhibitory Self-Attention (ISA), inspired by biological visual suppression mechanisms. ISA is the first self-attention variant to explicitly retain and leverage negative attention scores, circumventing the Softmax-induced constraint of non-negative outputs. By directly suppressing irrelevant features, ISA enhances the model’s selective focus on semantically meaningful regions. Integrated into the Vision Transformer architecture, ISA demonstrates consistent performance gains and improved out-of-distribution generalization across ImageNet-1k, COCO, and multiple robustness benchmarks. Attention visualizations further confirm that ISA produces sharper, more target-concentrated attention maps compared to standard self-attention.

Abstract

Vision Transformers (ViTs) have demonstrated remarkable performance in computer vision tasks. However, their self-attention mechanism often diffuses focus across background regions, relying on spurious correlations rather than object-relevant cues. Inspired by inhibitory mechanisms observed in biological vision systems, we propose the Inhibited Self-Attention (ISA), a novel self-attention that integrates inhibitory signals to enhance feature selectivity and suppress spurious responses. In contrast to conventional self-attention, which relies solely on positive attention values due to softmax normalization, our approach retains and utilizes negative attention scores to suppress irrelevant features and sharpen focus on objects of interest. Experiments across multiple datasets, including ImageNet-1k and COCO, and several robustness benchmarks demonstrate that ISA enhances object-centric selectivity, reduces shortcut reliance, and improves out-of-distribution generalization. Our analysis of relevance maps confirms that ViTs with ISA exhibit sharper, more localized focus on object-relevant regions while reducing distractions from non-relevant (background) features, enabling more reliable models. We release our code at https://github.com/prdvanderwal/inhibited-self-attention

Problem

• Vision Transformers
• self-attention
• spurious correlations
• object-centric focus
• background distraction

Innovation

• Inhibited Self-Attention
• Vision Transformers
• negative attention
• feature suppression
• object-centric focus