Cross-Layer Cache Aggregation for Token Reduction in Ultra-Fine-Grained Image Recognition

2024-12-31

AI Summary

To address the contradiction between high computational overhead of Vision Transformers (ViTs) and critical detail loss caused by token pruning in ultra-fine-grained image recognition (UFGIR), this paper proposes a cross-layer caching and aggregation mechanism. Methodologically, it introduces (1) the first cross-layer feature caching scheme, enabling deep-layer dynamic recovery and reuse of salient shallow-layer tokens; and (2) an attention-guided token reweighting aggregation module coupled with a plug-and-play classification head, ensuring compatibility with diverse ViT backbones and token pruning strategies. Extensive experiments across five UFGIR benchmarks, nine ViT architectures, and seven token reduction methods demonstrate that retaining only 10% of tokens achieves state-of-the-art accuracy—significantly breaking the performance bottleneck of fine-grained discrimination under extreme sparsification and substantially improving the accuracy-efficiency trade-off frontier.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Summarization

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Privacy-enhancing technologies
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Ultra-fine-grained image recognition (UFGIR) is a challenging task that involves classifying images within a macro-category. While traditional FGIR deals with classifying different species, UFGIR goes beyond by classifying sub-categories within a species such as cultivars of a plant. In recent times the usage of Vision Transformer-based backbones has allowed methods to obtain outstanding recognition performances in this task but this comes at a significant cost in terms of computation specially since this task significantly benefits from incorporating higher resolution images. Therefore, techniques such as token reduction have emerged to reduce the computational cost. However, dropping tokens leads to loss of essential information for fine-grained categories, specially as the token keep rate is reduced. Therefore, to counteract the loss of information brought by the usage of token reduction we propose a novel Cross-Layer Aggregation Classification Head and a Cross-Layer Cache mechanism to recover and access information from previous layers in later locations. Extensive experiments covering more than 2000 runs across diverse settings including 5 datasets, 9 backbones, 7 token reduction methods, 5 keep rates, and 2 image sizes demonstrate the effectiveness of the proposed plug-and-play modules and allow us to push the boundaries of accuracy vs cost for UFGIR by reducing the kept tokens to extremely low ratios of up to 10% while maintaining a competitive accuracy to state-of-the-art models. Code is available at: url{https://github.com/arkel23/CLCA}

Problem

• Fine-Grained Image Recognition
• Information Reduction
• Accuracy Degradation

Innovation

• Fine-Grained Image Recognition
• Information Point Reduction
• Computational Complexity Reduction