Spatial Priors via Space Filling Curves for Small and Limited Data Vision Transformers

2026-06-08

AI Summary

Vision Transformers lack explicit spatial inductive biases, which limits their performance in small-scale models or data-constrained settings. This work proposes VIOLIN, a lightweight masked attention mechanism that encodes spatial structural priors into the attention matrix using space-filling curves. VIOLIN introduces less than 0.0015% additional parameters and negligible computational overhead, remains compatible with parameter-efficient fine-tuning strategies such as LoRA, and integrates insights from linear Transformer masking and visual state space model scanning. Experimental results demonstrate that VIOLIN improves accuracy by up to 8.7% on VTAB-1K, achieves a 7.2% gain on pixel-level CIFAR-100 tasks, and consistently enhances the performance of small ViT architectures during ImageNet-1K pretraining.

Abstract

Though Vision Transformers (ViTs) have become the dominant backbone in many computer vision tasks, due to permutation equivariance, their attention mechanism lacks explicit spatial inductive biases. This become particularly important in two settings: when model capacity is small or training data is limited. Inspired by the attention masking strategies in Linear Transformers and the scanning patterns of Vision SSMs, we introduce VIOLIN, a lightweight masked attention mechanism that encodes spatial structure within attention via Space Filling Curves (SFCs) with less than 0.0015% extra parameters and negligible computational overhead. VIOLIN scans the image using multiple SFCs to construct curve-specific decay masks, which are then combined and multiplied with the attention matrix. Across a wide range of evaluations, VIOLIN consistently improves performance. In limited data regimes such as fine-tuning on VTAB-1K, it boosts accuracy across all task groups and by up to 8.7% on the tasks where spatial information is essential. It can be combined with parameter-efficient fine-tuning methods such as LoRA to further increase the performance. Beyond fine-tuning, VIOLIN improves various small scale ViT architectures (e.g., DeiT, DINO) during pretraining on ImageNet-1K. Additionally, on pixel-level CIFAR-100 training, a task that is highly dependent on location information, VIOLIN increases accuracy by up to 7.2%. Overall, VIOLIN provides a computationally efficient yet effective way to inject spatial inductive bias into ViTs, especially benefiting small models and limited data settings.

Problem

• Vision Transformers
• spatial inductive bias
• limited data
• small models
• attention mechanism

Innovation

• Space Filling Curves
• Vision Transformers
• Spatial Inductive Bias
• Masked Attention
• Limited Data