Vision Transformer Finetuning Benefits from Non-Smooth Components

2026-02-06

AI Summary

This work challenges the conventional assumption that smoother components are inherently superior by investigating the relationship between component adaptability and smoothness in Vision Transformers under transfer learning. To this end, it introduces “plasticity” as a novel metric quantifying a component’s sensitivity to input perturbations and systematically evaluates the plasticity of attention modules and feed-forward layers through both theoretical analysis and large-scale experiments. The study reveals that components with higher plasticity—i.e., lower smoothness—consistently yield better fine-tuning performance and significantly enhance downstream task accuracy. These findings establish plasticity as a critical lens for assessing adaptation capacity and provide a principled guideline for selecting components in efficient fine-tuning, thereby overturning the long-standing design paradigm centered on smoothness.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

The smoothness of the transformer architecture has been extensively studied in the context of generalization, training stability, and adversarial robustness. However, its role in transfer learning remains poorly understood. In this paper, we analyze the ability of vision transformer components to adapt their outputs to changes in inputs, or, in other words, their plasticity. Defined as an average rate of change, it captures the sensitivity to input perturbation; in particular, a high plasticity implies low smoothness. We demonstrate through theoretical analysis and comprehensive experiments that this perspective provides principled guidance in choosing the components to prioritize during adaptation. A key takeaway for practitioners is that the high plasticity of the attention modules and feedforward layers consistently leads to better finetuning performance. Our findings depart from the prevailing assumption that smoothness is desirable, offering a novel perspective on the functional properties of transformers. The code is available at https://github.com/ambroiseodt/vit-plasticity.

Problem

• Vision Transformer
• transfer learning
• smoothness
• plasticity
• finetuning

Innovation

• plasticity
• non-smoothness
• vision transformer
• fine-tuning
• transfer learning