On the Performance of Differentially Private Optimization with Heavy-Tail Class Imbalance

2025-07-14

AI Summary

This work addresses the significant performance degradation of differentially private optimization algorithms—such as DP-GD—on infrequent classes under heavy-tailed, class-imbalanced data distributions. We propose DP-AdamBC, a novel differentially private optimizer that incorporates unbiased second-order curvature estimation to adaptively adjust gradient update directions, thereby mitigating the ill-conditioning of the loss landscape induced by class imbalance. Comprehensive evaluations on controlled benchmarks and real-world datasets demonstrate that DP-AdamBC improves training accuracy on the least-frequent class by approximately 8% and 5%, respectively, outperforming standard DP-GD. To our knowledge, this is the first work to integrate unbiased curvature estimation into the differentially private optimization framework. Our approach advances fair and robust private learning for heavy-tailed and long-tailed classification tasks, offering both theoretical insight and practical utility.

Technology Category

• Machine Learning: Privacy
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Non-convex Optimization

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

In this work, we analyze the optimization behaviour of common private learning optimization algorithms under heavy-tail class imbalanced distribution. We show that, in a stylized model, optimizing with Gradient Descent with differential privacy (DP-GD) suffers when learning low-frequency classes, whereas optimization algorithms that estimate second-order information do not. In particular, DP-AdamBC that removes the DP bias from estimating loss curvature is a crucial component to avoid the ill-condition caused by heavy-tail class imbalance, and empirically fits the data better with $approx8%$ and $approx5%$ increase in training accuracy when learning the least frequent classes on both controlled experiments and real data respectively.

Problem

• Analyzing private learning optimization under heavy-tail imbalance
• Comparing DP-GD and second-order methods for low-frequency classes
• Proposing DP-AdamBC to improve accuracy in imbalanced data

Innovation

• DP-AdamBC removes DP bias for better accuracy
• Second-order optimization handles heavy-tail imbalance
• Improves training accuracy by 5-8%