REDI: Corpus Aware Patch Ranking for DINOv3 Token Reduction

2026-06-30

AI Summary

This work proposes the REDI scoring mechanism for optimal image patch token allocation in Vision Transformers under a fixed token budget. REDI uniquely integrates supervised TF-IDF—derived from category-conditioned visual word statistics across a corpus—with image-specific attention maps to assess patch importance. Implemented within the DINOv3 ViT-B/16 architecture and combined with visual vocabulary quantization and a token reduction operator, REDI achieves a 46.8% sequence length reduction on ImageNet-1K (from 201 to 107 tokens) while attaining a Top-1 accuracy of 84.706%. This performance significantly surpasses baselines relying solely on attention or TF-IDF, demonstrating the complementary nature of corpus-level statistical cues and instance-specific attention signals in effective token prioritization.

Abstract

Most token reduction methods for Vision Transformers seek favorable tradeoffs between accuracy and efficiency by pruning, merging, or pooling patch tokens. REDI (Relevance for DINOv3 Token Reduction) studies this question through a controlled supervised reference: how should a fixed token budget be allocated across patches for image classification? REDI quantizes final block DINOv3 patch representations into a visual vocabulary and derives class conditioned corpus scores using supervised TF-IDF over visual words. For each validation image, the ground truth class selects a row of the TF-IDF table, and four transformed views produce a TF-IDF map aligned to a reference center crop. A separate dense pass on the same crop provides an attention map. After independent min max normalization, their elementwise product defines the REDI score. A fixed keep, merge, and compress operator then uses score rank to assign patch roles and score magnitude to weight merging and compression. With precomputed REDI scores, a frozen DINOv3 ViT-B/16 backbone, and the same linear classifier used for dense evaluation, the operator reduces the sequence length from 201 to 107 tokens, a 46.8% sequence reduction. The REDI variant based on incoming attention mass achieves 84.706% Top-1 accuracy on ImageNet-1K, compared with 83.514% for the dense baseline, 82.634% for incoming attention mass alone, and 81.796% for supervised TF-IDF alone. The same corpus term also improves reduced classification for three alternative attention formulations relative to their attention only counterparts. Together, these controlled comparisons indicate that class specific corpus statistics and image specific attention provide complementary signals for patch ranking in this setting.

Problem

• token reduction
• Vision Transformers
• image classification
• patch ranking
• DINOv3

Innovation

• token reduction
• corpus-aware
• supervised TF-IDF
• attention map
• Vision Transformer