Active Spatial Guidance: Eliminating Injected Positional Mechanisms in Vision Transformers

2026-07-01

AI Summary

Vision Transformers (ViTs) require explicit position encodings to compensate for the permutation invariance of self-attention, which compromises architectural simplicity and generalization. This work proposes Active Spatial Guidance, a training strategy that introduces an auxiliary 2D coordinate regression loss during training to implicitly encourage the model to learn spatial structure without any explicit position encoding; at inference, only the clean ViT backbone is retained. The method demonstrates for the first time that spatial inductive bias in ViTs can be implicitly acquired through supervisory signals and is compatible with any ViT architecture. Using a DINOv3 backbone, it consistently outperforms strong baselines across ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim depth estimation, while significantly enhancing robustness to resolution shifts.

Abstract

Vision Transformers (ViTs) commonly rely on injected positional mechanisms to address self-attention's permutation invariance. Motivated by the spatial regularities of natural images, we ask whether spatial organization can be induced from data rather than explicitly injected. Under controlled, matched from-scratch training, we propose Active Spatial Guidance (Guidance), a training-only objective that disables positional injection and applies an auxiliary 2D coordinate-regression loss to the final-layer patch tokens. The guidance head is used only during training and removed for inference; the deployed model consists of a positional-injection-free ViT encoder and the task-specific prediction module. Using DINOv3 ViT backbones, Guidance consistently improves performance on ImageNet-100 classification, ADE20K semantic segmentation, and Hypersim monocular depth estimation, outperforming strong injected baselines such as learned absolute positional embeddings and rotary positional embeddings under identical training protocols. On ImageNet-100, broader comparisons against representative injected positional designs further support Guidance's effectiveness. Guidance also improves robustness under resolution transfer, and multi-resolution training further strengthens accuracy across input sizes. Overall, our results suggest that spatial inductive bias in ViTs need not be architecturally injected, but can be shaped through training-time supervision. The code used for training and evaluation is publicly available in https://github.com/cloudlc/asg.

Problem

• Vision Transformers
• positional mechanisms
• spatial inductive bias
• self-attention
• permutation invariance

Innovation

• Active Spatial Guidance
• Vision Transformers
• positional injection
• coordinate regression
• inductive bias