Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

2026-07-01

AI Summary

This work addresses the inefficiency of vision transformers (ViTs) during inference, which stems from uniform computation across all image regions, leading to redundancy and poor energy efficiency. While existing adaptive acceleration methods often suffer from representation instability and accuracy degradation when combining multiple mechanisms, this paper proposes Fusion—a framework that sequentially integrates token merging, early exiting, and pruning in a staged manner. It first merges redundant tokens, then employs a lightweight routing module to assess prediction confidence, applying pruning only to low-confidence samples to ensure complementary rather than conflicting interactions among mechanisms. This design enables input-adaptive compression and allows flexible, training-free trade-offs between accuracy and latency at inference time. On ImageNet-1k, Fusion matches or surpasses state-of-the-art adaptive ViT methods, reducing calibration error by up to 4× and inference energy consumption by 48% under comparable computational costs, while demonstrating strong generalization across diverse datasets and ViT backbones.

Abstract

Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to $4\times$ and inference energy by $48\%$. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.

Problem

• Vision Transformers
• adaptive inference
• token compression
• early exiting
• accuracy degradation

Innovation

• adaptive inference
• token merging
• early exiting
• vision transformers
• dynamic computation