Accelerating Vision Transformers with Adaptive Patch Sizes

2025-10-20

AI Summary

Vision Transformers (ViTs) suffer from severely degraded training and inference efficiency on high-resolution images due to fixed-size patching, which yields excessively long sequences. This work proposes a content-aware adaptive multi-scale patching method that dynamically assigns fine- and coarse-grained patches within a single image based on local complexity—requiring no architectural modifications or model retraining for integration with existing ViT models. To our knowledge, this is the first approach enabling heterogeneous patch partitioning *within* individual images. Evaluated on ViT-L and ViT-H, it improves throughput by 40% and 50%, respectively, and accelerates both training and inference by up to 30% on high-resolution dense prediction tasks—without any accuracy loss. The core contribution is a lightweight, plug-and-play patching paradigm that jointly optimizes computational efficiency and representational capacity.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Vision Transformers (ViTs) partition input images into uniformly sized patches regardless of their content, resulting in long input sequence lengths for high-resolution images. We present Adaptive Patch Transformers (APT), which addresses this by using multiple different patch sizes within the same image. APT reduces the total number of input tokens by allocating larger patch sizes in more homogeneous areas and smaller patches in more complex ones. APT achieves a drastic speedup in ViT inference and training, increasing throughput by 40% on ViT-L and 50% on ViT-H while maintaining downstream performance, and can be applied to a previously fine-tuned ViT, converging in as little as 1 epoch. It also significantly reduces training and inference time without loss of performance in high-resolution dense visual tasks, achieving up to 30% faster training and inference in visual QA, object detection, and semantic segmentation.

Problem

• Reducing input sequence length in Vision Transformers
• Optimizing patch allocation based on image complexity
• Accelerating ViT training and inference while maintaining performance

Innovation

• Using adaptive patch sizes for different image regions
• Reducing input tokens with larger patches in homogeneous areas
• Maintaining performance while accelerating training and inference