SAC-ViT: Semantic-Aware Clustering Vision Transformer with Early Exit

2025-02-27

AI Summary

To address the high computational overhead of Vision Transformers (ViTs) on resource-constrained devices—stemming from the quadratic complexity of self-attention—the paper proposes an end-to-end trainable, two-stage semantic-aware inference architecture. In Stage I, an early-exit mechanism generates coarse predictions and localizes salient image regions. In Stage II, a novel non-iterative, semantic-aware dynamic token clustering (SAC) method models target regions at fine granularity while reusing non-target tokens to eliminate redundant computation. The approach integrates region cropping, embedding remapping, and local attention to substantially reduce computational load. Evaluated on the DeiT benchmark, it achieves a 62% reduction in FLOPs and a 1.98× throughput improvement, with zero accuracy degradation. This work establishes a new paradigm for efficient ViT deployment.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

The Vision Transformer (ViT) excels in global modeling but faces deployment challenges on resource-constrained devices due to the quadratic computational complexity of its attention mechanism. To address this, we propose the Semantic-Aware Clustering Vision Transformer (SAC-ViT), a non-iterative approach to enhance ViT's computational efficiency. SAC-ViT operates in two stages: Early Exit (EE) and Semantic-Aware Clustering (SAC). In the EE stage, downsampled input images are processed to extract global semantic information and generate initial inference results. If these results do not meet the EE termination criteria, the information is clustered into target and non-target tokens. In the SAC stage, target tokens are mapped back to the original image, cropped, and embedded. These target tokens are then combined with reused non-target tokens from the EE stage, and the attention mechanism is applied within each cluster. This two-stage design, with end-to-end optimization, reduces spatial redundancy and enhances computational efficiency, significantly boosting overall ViT performance. Extensive experiments demonstrate the efficacy of SAC-ViT, reducing 62% of the FLOPs of DeiT and achieving 1.98 times throughput without compromising performance.

Problem

• Reduces Vision Transformer computational complexity for resource-constrained devices.
• Enhances ViT efficiency via Semantic-Aware Clustering and Early Exit stages.
• Improves performance by reducing spatial redundancy and optimizing attention mechanism.

Innovation

• Early Exit stage reduces computational load
• Semantic-Aware Clustering enhances token efficiency
• End-to-end optimization boosts ViT performance