Subtoken Vision Transformer for Fine-grained Recognition

2026-07-10

AI Summary

Standard Vision Transformers compress image patches into single tokens, limiting their ability to capture subtle distinctions in fine-grained recognition tasks. To address this, this work proposes the Subtoken Vision Transformer (SubViT), which enhances local representation by assigning multiple subtokens to discriminative image regions while preserving the original token for global context. A novel lightweight per-image router is introduced to predict token importance without additional forward computation. The model is trained in two stages, leveraging attention map sampling, feature degradation distance estimation, and knowledge distillation. Evaluated on CUB, FGVC-Aircraft, and Stanford Cars, SubViT improves the average accuracy of DINOv2 on novel categories from 81.3% to 84.7%, with only a 0.50ms latency increase and 3.4% more FLOPs—achieving a 73.8% reduction in latency compared to Retina Patch.

Abstract

We present Subtoken Vision Transformer (SubViT), a selective image tokenization method for fine-grained visual recognition. Standard Vision Transformers compress each fixed-size patch into a single token, although fine-grained distinctions often depend on localized variations within only a few patches. SubViT addresses this mismatch by representing discriminative patches with multiple subtokens while retaining the original token sequence for global context, thereby allocating additional capacity where it is most needed. Since attention heads encode complementary semantics and extracting attention maps at inference requires an extra backbone forward, we adopt a two-stage training strategy. Stage 1 fine-tunes the ViT using subdivision regions sampled from random attention heads, exposing the model to diverse subdivision patterns. Stage 2 identifies informative attention maps through feature-degradation distances and distills them into a lightweight single-map router, which directly predicts deterministic token-importance scores without a separate attention forward. We evaluate SubViT on Generalized Category Discovery (GCD), a challenging task requiring both fine-grained discrimination and generalization to unlabeled novel categories. Across CUB, FGVC-Aircraft, and Stanford Cars, SubViT improves the average novel-category accuracy of DINOv2 from $81.3\%$ to $84.7\%$, with only $0.50$ ms additional latency and $3.4\%$ more FLOPs, while reducing latency by $73.8\%$ relative to Retina Patch. Results on CIFAR-10 and ImageNet-100 demonstrate its broader applicability.

Problem

• fine-grained recognition
• Vision Transformer
• tokenization
• local variation
• discriminative patches

Innovation

• Subtoken Vision Transformer
• fine-grained recognition
• selective tokenization
• attention distillation
• Generalized Category Discovery