🤖 AI Summary
This work addresses the limited exploration of gradient perturbations during backpropagation in existing training methods, which often focus solely on forward perturbations and lack mechanisms for class-aware adaptive optimization. To bridge this gap, we establish a unified theoretical framework for gradient perturbation that subsumes techniques such as Sharpness-Aware Minimization (SAM) and gradient clipping. Within this framework, we propose Label-aware Perturbed Gradients (LPG), a novel method that adaptively modulates gradient norms to induce class-conditional augmentation effects. Leveraging PAC-Bayesian analysis, we theoretically connect the magnitude of gradient perturbations to generalization performance. LPG is designed as a modular, plug-and-play component and demonstrates consistent superiority over existing approaches across balanced classification, long-tailed learning, and noisy-label settings, while seamlessly integrating with other training strategies.
📝 Abstract
Deep neural network training involves both forward propagation (from features through logits to loss) and backward propagation (from loss through gradients to parameter updates). While perturbations along the forward chain, including feature perturbation, logit perturbation, and label perturbation, have been extensively studied, the backward chain's gradient perturbation has received little systematic investigation. In this paper, we establish a unified framework for gradient perturbation, revealing that existing methods such as Sharpness-Aware Minimization (SAM), gradient clipping, and gradient noise injection can all be interpreted as imposing specific forms of gradient perturbation. Analogous to the recently proposed Logit Perturbation Learning (LPL), we conjecture that amplifying the gradient norm for a class acts as positive augmentation (enhancing learning), while dampening it acts as negative augmentation (suppressing overfitting). Based on these observations, we propose Learning to Perturb Gradients (LPG), which adaptively perturbs logit-level gradients at the class level to achieve category-aware training. We also establish theoretical connections between gradient perturbation bounds and generalization guarantees via PAC-Bayesian analysis. Experiments on balanced classification, long-tail classification, and noisy label learning demonstrate that LPG consistently outperforms existing methods and can be combined with them as a plug-in module.