🤖 AI Summary
Existing deep learning approaches lack a unified and effective mechanism for perturbing hidden-layer activations, limiting their ability to systematically enhance model generalization. This work proposes a class-aware, learnable perturbation framework for activations (LPA), which, for the first time, systematically analyzes the underlying mechanisms of hidden activation perturbation and reveals the opposing effects of expansive versus contractive perturbations on generalization. By leveraging projected gradient descent (PGD) optimization, LPA enables class-adaptive perturbations and theoretically connects them to flat minima and inter-layer amplification effects. Extensive experiments demonstrate that LPA significantly outperforms current methods across balanced classification, long-tailed learning, and domain generalization tasks, and further exhibits complementary gains when combined with logit-level perturbation techniques such as LPL.
📝 Abstract
Deep neural networks process data through a cascade of representations: input features, hidden activations, logits, and loss. While perturbations at the input, logit, and label levels have been systematically studied, the intermediate hidden activations, which constitute the bulk of the network's computation, have received no unified perturbation analysis. In this paper, we establish a unified framework for hidden activation perturbation, revealing that Dropout, Manifold Mixup, adversarial feature perturbation, and related methods all impose specific forms of activation perturbation but with class-agnostic or random strategies. We conjecture that expansive perturbation (increasing activation norm) acts as positive augmentation, while contractive perturbation (decreasing activation norm) acts as negative augmentation, and that the perturbation layer determines whether the effect resembles input-level augmentation (shallow layers) or logit-level manipulation (deep layers). We propose Learning to Perturb Activations (LPA), which adaptively perturbs activations at a selected hidden layer with class-level perturbations learned via PGD. We further provide theoretical analysis connecting activation perturbation to flat minima and perturbation amplification through layers. Experiments on balanced classification, long-tail classification, and domain generalization demonstrate that LPA consistently outperforms existing methods and provides complementary benefits to logit perturbation methods such as LPL.