DnA: Denoising Attention for Visual Tasks

2026-06-25

AI Summary

Standard Softmax attention in vision tasks is susceptible to noise, which can degrade the representation of critical features. This work proposes Denoising Attention (DnA), introducing for the first time a cooperative positive-negative query mechanism coupled with a dual-subspace separation strategy. Specifically, positive and negative queries capture relevant and interfering features, respectively, and project them into two orthogonal subspaces with maximally separated principal angles, thereby enhancing feature discriminability and suppressing attention noise. Integrated into a Vision Transformer (ViT) architecture, DnA achieves a 0.8% accuracy gain on ImageNet-1K, improves performance by 1.8% on video understanding benchmarks, and yields a 0.5% improvement in video-language foundation models.

Abstract

The softmax activation in multihead attention (MHA) is the de facto standard for attention-based models in visual perception tasks. However, standard softmax can produce noisy attention patterns that dilute relevant features and degrade its performance. In this paper, we propose Denoising Attention or DnA, in which, first, a positive query identifies which image features belong to the correct class, and a negative query identifies closely associated but irrelevant image features. DnA then projects these interactions into two distinct subspaces with larger principal angles, promoting subspace separation and improved discriminability. Using a ViT-B backbone, our proposed DnA achieves an absolute gain of 0.8% on ImageNet-1K compared to the baseline. We further show improvements across multiple visual understanding tasks, including video understanding with video transformers (1.8%) and video LLMs (0.5%). Our extensive empirical analyses justify the design choices involving two interacting subspaces and the denoising effect of DnA.

Problem

• multihead attention
• softmax
• attention noise
• visual tasks
• feature dilution

Innovation

• Denoising Attention
• multihead attention
• subspace separation
• visual transformers
• attention denoising