Vertical Fusion: Condensing Internal Representations for Robust ViT Classification

2026-07-11

AI Summary

Vision Transformers (ViTs) typically rely solely on their final-layer features for classification, overlooking the discriminative information embedded in intermediate layers that could correct prediction errors. This work systematically evaluates the classification capability of each intermediate layer in ViTs, uncovering a correspondence between redundancy and correctness within the model’s internal representations. To leverage this insight, the authors propose VFusion, a vertical fusion mechanism that efficiently aggregates multi-level features into a unified latent space via learnable low-dimensional mappings. VFusion replaces costly horizontal ensembles with a single-model solution and achieves substantial improvements over existing baselines under both in-distribution and out-of-distribution settings. It closes 45% of the accuracy gap between the best individual layer and the theoretical oracle, demonstrating consistent performance across varying model scales and pretraining configurations.

Abstract

Despite exposing rich intermediate representations, Vision Transformers (ViTs) are almost exclusively utilized as black-box feature extractors, where only the last layer is considered for downstream tasks. We challenge this convention by introducing the notion of recoverability: the capacity of intermediate representations to correct last-layer failures. By evaluating independent classification probes at every model depth across 16 datasets, we observe that intermediate probes correctly classify 18% to 76% of samples that the last-layer probe misclassifies. We show that these gains are not primarily driven by predictive diversity, but by a redundancy-correctness correspondence, where the internal hierarchy acts as a series of stable, redundant probes of a shared discriminative signal. While established horizontal ensemble strategies (i.e., across multiple models) can improve performance, they incur high computational cost and ignore this vertical signal within a single model. To bridge this gap, we propose VFusion, a principled vertical aggregation strategy employing a learnable mapping into a low-dimensional latent space that synthesizes features across the internal ViT hierarchy. VFusion substantially outperforms established aggregation baselines in both in-distribution and out-of-distribution settings, notably closing 45% of the accuracy gap between the best individual layer and a theoretical oracle performance. Our gains consistently generalize across model sizes and pre-training regimes, confirming that VFusion offers a robust and efficient alternative to horizontal ensemble methods. The code is available at https://github.com/francescodisalvo05/vit-vertical-fusion.

Problem

• Vision Transformers
• intermediate representations
• vertical fusion
• classification robustness
• representation redundancy

Innovation

• Vertical Fusion
• Vision Transformers
• Intermediate Representations
• Robust Classification
• Feature Aggregation