Reducing Class-Wise Performance Disparity via Margin Regularization

2026-01-30

AI Summary

Despite balanced training data across classes, deep neural networks often exhibit significant inter-class performance disparities. To address this issue, this work proposes MR², a method that dynamically adjusts class-dependent margins in both the logit and representation spaces to refine decision boundaries for difficult classes and enhance intra-class compactness. Building upon a derived class-sensitive generalization bound, the approach reveals how feature variability influences classification error and leverages this insight to design a dynamic margin regularization mechanism. This mechanism integrates adaptive logit margin adjustment, representation margin penalization, and a measure of class-specific feature dispersion. Evaluated on seven datasets including ImageNet, MR² consistently improves overall accuracy and performance on hard classes without compromising easy-class accuracy, effectively narrowing the performance gap across classes.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Computer Vision: Representation Learning for Vision
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Deep neural networks often exhibit substantial disparities in class-wise accuracy, even when trained on class-balanced data, posing concerns for reliable deployment. While prior efforts have explored empirical remedies, a theoretical understanding of such performance disparities in classification remains limited. In this work, we present Margin Regularization for Performance Disparity Reduction (MR$^2$), a theoretically principled regularization for classification by dynamically adjusting margins in both the logit and representation spaces. Our analysis establishes a margin-based, class-sensitive generalization bound that reveals how per-class feature variability contributes to error, motivating the use of larger margins for hard classes. Guided by this insight, MR$^2$ optimizes per-class logit margins proportional to feature spread and penalizes excessive representation margins to enhance intra-class compactness. Experiments on seven datasets, including ImageNet, and diverse pre-trained backbones (MAE, MoCov2, CLIP) demonstrate that MR$^2$ not only improves overall accuracy but also significantly boosts hard class performance without trading off easy classes, thus reducing performance disparity. Code is available at: https://github.com/BeierZhu/MR2

Problem

• class-wise performance disparity
• deep neural networks
• classification
• margin regularization
• generalization

Innovation

• margin regularization
• class-wise performance disparity
• generalization bound
• feature variability
• intra-class compactness