Advancing Vision Transformer with Enhanced Spatial Priors

2026-04-20

AI Summary

This work addresses the challenges faced by Vision Transformers (ViTs) in efficiently modeling spatial relationships due to the absence of explicit spatial priors and the high computational complexity of self-attention. To overcome these limitations, the authors propose a spatial prior modeling approach based on Euclidean distance decay, which enhances the model’s awareness of local geometric structure. Additionally, they introduce a spatial-agnostic grouped attention mechanism that replaces conventional factorization strategies, reducing computational overhead while improving representational flexibility. The proposed method achieves a top-1 accuracy of 86.6% on ImageNet-1k and demonstrates strong performance across multiple vision tasks, including image classification, object detection, instance segmentation, and semantic segmentation.

Technology Category

• Computer Vision: Large Vision Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

In recent years, the Vision Transformer (ViT) has garnered significant attention within the computer vision community. However, the core component of ViT, Self-Attention, lacks explicit spatial priors and suffers from quadratic computational complexity, limiting its applicability. To address these issues, we have proposed RMT, a robust vision backbone with explicit spatial priors for general purposes. RMT utilizes Manhattan distance decay to introduce spatial information and employs a horizontal and vertical decomposition attention method to model global information. Building on the strengths of RMT, Euclidean enhanced Vision Transformer (EVT) is an expanded version that incorporates several key improvements. Firstly, EVT uses a more reasonable Euclidean distance decay to enhance the modeling of spatial information, allowing for a more accurate representation of spatial relationships compared to the Manhattan distance used in RMT. Secondly, EVT abandons the decomposed attention mechanism featured in RMT and instead adopts a simpler spatially-independent grouping approach, providing the model with greater flexibility in controlling the number of tokens within each group. By addressing these modifications, EVT offers a more sophisticated and adaptable approach to incorporating spatial priors into the Self-Attention mechanism, thus overcoming some of the limitations associated with RMT and further enhancing its applicability in various computer vision tasks. Extensive experiments on Image Classification, Object Detection, Instance Segmentation, and Semantic Segmentation demonstrate that EVT exhibits exceptional performance. Without additional training data, EVT achieves 86.6% top1-acc on ImageNet-1k.

Problem

• Vision Transformer
• Self-Attention
• spatial priors
• computational complexity
• computer vision

Innovation

• Vision Transformer
• spatial priors
• Euclidean distance decay
• attention mechanism
• token grouping