🤖 AI Summary
Empirical Risk Minimization (ERM) suffers from degraded generalization under long-tailed class distributions. Method: This paper proposes a data-dependent shrinkage technique and establishes the first fine-grained, class-aware unified generalization upper bound. Unlike conventional coarse-grained analyses relying on global statistics, our bound explicitly quantifies how class-specific terms influence generalization error. Contribution/Results: The bound provides the first systematic theoretical explanation of the intrinsic mechanisms underlying reweighting and logit adjustment—resolving several counterintuitive empirical observations. Leveraging this theory, we design a principled learning algorithm that significantly improves minority-class accuracy on standard long-tailed benchmarks—including CIFAR-10-LT and ImageNet-LT—outperforming state-of-the-art methods.
📝 Abstract
Real-world datasets are typically imbalanced in the sense that only a few classes have numerous samples, while many classes are associated with only a few samples. As a result, a na""ive ERM learning process will be biased towards the majority classes, making it difficult to generalize to the minority classes. To address this issue, one simple but effective approach is to modify the loss function to emphasize the learning on minority classes, such as re-weighting the losses or adjusting the logits via class-dependent terms. However, existing generalization analysis of such losses is still coarse-grained and fragmented, failing to explain some empirical results. To bridge this gap, we propose a novel technique named data-dependent contraction to capture how these modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment in a unified manner. Furthermore, a principled learning algorithm is developed based on the theoretical insights. Finally, the empirical results on benchmark datasets not only validate the theoretical results but also demonstrate the effectiveness of the proposed method.
