🤖 AI Summary
This work addresses the limitation of prevailing vision foundation models that rely on fixed-grid tokenization at the input stage, overlooking the potential for content-adaptive optimization. The authors propose a parameter-free, input-adaptive tokenizer that dynamically determines the identity, position, scale, and number of tokens based on local visual entropy before the backbone network. Instead of standard ViT grid patches, it generates multi-scale, spiral-ring patches centered on content-rich regions. This approach establishes input-adaptive tokenization as a third lever—alongside model architecture and inference algorithms—for resource-efficient reasoning, enabling content-aware tokenization and precise allocation of computational resources. Experiments on four fine-grained benchmarks show a 60% reduction in input tokens, an 84% decrease in per-layer self-attention FLOPs, 18–29% higher throughput, and consistent accuracy gains of 1.7–2.1 percentage points.
📝 Abstract
Most adaptive-inference techniques for foundation models change what the model does - early exit, MoE routing, KV-cache compression, dynamic attention sparsity. The input that hits the backbone, however, remains a fixed-grid tokenisation indifferent to image content. We argue that this is a missed lever. We present SpiralFovea, a parameter-free, input-adaptive tokeniser in which token identity, location, scale, and count are all functions of local visual entropy and selection completes before any backbone parameter is queried. Around content-driven hotspot anchors, multi-scale spiral rings produce <= 78 patches that replace the standard 196-patch ViT grid at the input stage. Across four canonical fine-grained benchmarks, SpiralFovea yields +1.7-2.1 pp accuracy with a 60% reduction in input tokens, an 84% reduction in self-attention FLOPs at every transformer layer, and 18-29% throughput gains over the matched static tokenisation baseline. A controlled ablation on CUB-200-2011 Genus across four backbones reveals a clean diagnostic: the gain magnitude tracks inversely with the strength of the backbone's whole-image positional prior, isolating self-supervised foundation models as the regime where input-adaptive tokenisation is most valuable.