When Token Compression Breaks: Structural Pruning vs. Token Reduction for Robust ViT Segmentation under High Compression

2026-07-02

AI Summary

This work addresses the high computational cost of Vision Transformers (ViTs) in semantic segmentation and the unclear robustness of mainstream compression methods under high compression ratios and corrupted inputs. For the first time, it systematically compares structural pruning and token compression under a unified protocol at matched computational budgets, revealing that token compression suffers severe accuracy degradation at high compression rates, whereas structural pruning exhibits greater stability. Building on this insight, the authors propose a joint compression strategy that first applies structural pruning followed by token merging. Extensive experiments on ADE20K, Cityscapes, and their corruption variants demonstrate that this approach significantly outperforms individual compression techniques under high compression, achieving a superior trade-off between accuracy, computational efficiency, and robustness.

Abstract

Vision Transformers (ViTs) are strong backbones for semantic segmentation, but their computational cost limits deployment. Recent token compression methods for efficient transformer-based segmentation reduce this cost by decreasing the number of tokens. However, existing evaluations primarily focus on low-to-moderate compression, leaving their behavior under aggressive compression and corrupted inputs unclear. Meanwhile, structural pruning provides an orthogonal route to efficiency by removing redundant components in the ViT architecture, but is rarely compared to token compression under a unified protocol. To bridge this gap, we benchmark representative token compression and structural pruning methods for ViT-based semantic segmentation under matched FLOPs on ADE20K and Cityscapes, together with their common-corruption variants ADE20K-C and Cityscapes-C. Our results reveal a consistent trend on both clean and corrupted inputs: token compression is highly effective at mild reductions but degrades sharply when compression becomes severe, consistent with substantial information loss from overly aggressive token reduction. In contrast, structural pruning exhibits a smoother degradation curve and is more stable at high compression. Motivated by these findings, we study a prune-then-merge pipeline that applies moderate token compression on top of a moderately pruned backbone. At comparable FLOPs, this combined strategy consistently achieves a better accuracy-robustness trade-off at high compression, offering a practical recipe for deployment-oriented ViT segmentation. Code is available at https://github.com/phatnguyencs/vit-seg-compression.

Problem

• Vision Transformers
• semantic segmentation
• token compression
• structural pruning
• robustness

Innovation

• token compression
• structural pruning
• Vision Transformers
• semantic segmentation
• model robustness