Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers

2026-07-10

AI Summary

This work addresses the trade-off between efficiency and robustness in Vision Transformers by proposing a dynamic token selection architecture inspired by the human visual system. For the first time, it integrates foveated guidance and foveal sampling mechanisms into Vision Transformers: a fixation module identifies salient image regions, while a foveal module generates multi-scale embeddings and adaptively prunes redundant tokens. This approach simultaneously enhances model efficiency and intrinsic robustness without requiring additional adversarial training. Experimental results demonstrate that, using only 50% of the original token budget, the proposed model achieves 81.9% top-1 accuracy on ImageNet—surpassing DeiT-S—while reducing multiply-add operations by 34.57%.

Abstract

The human visual system (HVS) employs foveated sampling and eye movements to achieve efficient perception, conserving both metabolic energy and computational resources. Drawing inspiration from this robustness and adaptability, we introduce the Foveated Dynamic Transformer (FDT), a foveation-guided dynamic token-selection architecture that integrates these mechanisms into a vision transformer framework. The FDT exhibits strong resilience to various types of noise and adversarial attacks, despite not being explicitly trained for such challenges. This inherent robustness is achieved through the use of fixation and foveation modules: the fixation module identifies fixation points to filter out irrelevant information, while the foveation module generates foveated embeddings with multi-scale information. At the 50% fixation-budget setting, FDT achieves higher accuracy than DeiT-S (81.9% vs. 80.9%) while reducing multiply-accumulate operations by 34.57%, highlighting one operating point on its accuracy-efficiency trade-off. These attributes position FDT as an HVS-inspired step toward artificial neural networks that combine adaptive computation with improved resilience.

Problem

• foveation
• dynamic token selection
• vision transformers
• robustness
• efficiency

Innovation

• Foveation
• Dynamic Token Selection
• Vision Transformer
• Robustness
• Efficient Computation