Rethinking Depth Pruning for Vision Transformers: A Heterogeneity-Aware Perspective

2026-07-04

AI Summary

Existing depth pruning methods for vision transformers often overlook inter-layer heterogeneity, leading to significant accuracy degradation upon full-layer removal and thus limiting achievable acceleration. To address this, this work proposes HetDPT, the first depth pruning framework that explicitly accounts for inter-layer heterogeneity. HetDPT introduces a fine-tuning-free, heterogeneity-aware pruning strategy that effectively circumvents dimension mismatch issues and seamlessly integrates with width pruning, making it compatible with mainstream ViT architectures such as DeiT. On ImageNet, HetDPT achieves a 1.58× speedup on DeiT-B without any accuracy loss; when combined with width pruning, it attains an impressive 5.19× acceleration under extreme compression, substantially outperforming existing approaches.

Abstract

While prior studies have successfully compressed vision Transformers (ViTs) through various pruning techniques, most have concentrated on width pruning to achieve significant reductions in model size. Depth pruning, which removes entire layers from a ViT, is notoriously difficult for accuracy recovery despite its potential to deliver higher speedups, limiting the acceleration achieved by existing joint width-and-depth pruning methods. In this work, we reveal that the failure of existing depth pruning methods lies in their neglect of heterogeneity between different layers, and we introduce HetDPT, a heterogeneity-aware depth pruning method that avoids dimension mismatch. Comprehensive experiments on ImageNet-1K, CIFAR-100, COCO, and ADE20K validate our method: HetDPT achieves a 1.58$\times$ speedup for DeiT-B while maintaining accuracy and a 1.39$\times$ speedup for DeiT-S with nearly no accuracy degradation. Furthermore, when combined with width pruning, HetDPT+ sets a new state-of-the-art record in extreme ViT pruning, enhancing the acceleration ratio from 4.24$\times$ to 5.19$\times$ for the Isomorphic-Pruning-2.6G configuration while maintaining near-lossless accuracy; our code is available at https://github.com/Efficient-AI-for-All/HetDPT.

Problem

• Depth Pruning
• Vision Transformers
• Heterogeneity
• Model Compression
• Accuracy Recovery

Innovation

• Depth Pruning
• Vision Transformers
• Heterogeneity-Aware
• Model Compression
• Acceleration