Harmonized Gradient Descent for Class Imbalanced Data Stream Online Learning

2025-08-15

AI Summary

Real-world data streams often exhibit skewed class distributions, severely degrading minority-class performance in online learning. To address this, we propose the Harmonized Gradient Descent (HGD) framework—the first approach to mitigate class imbalance by explicitly equalizing gradient norms across classes. HGD requires no data buffering, auxiliary parameters, or prior knowledge, ensuring strong generality and plug-and-play applicability. At each update step, it dynamically rescales the gradient norm of each sample according to its class, thereby harmonizing optimization step sizes across classes. We theoretically establish a sublinear regret bound for HGD. Extensive experiments on multiple imbalanced data stream benchmarks demonstrate that HGD significantly improves minority-class accuracy (average gain +12.7%) while preserving overall accuracy and convergence stability—outperforming state-of-the-art resampling and cost-sensitive methods.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Many real-world data are sequentially collected over time and often exhibit skewed class distributions, resulting in imbalanced data streams. While existing approaches have explored several strategies, such as resampling and reweighting, for imbalanced data stream learning, our work distinguishes itself by addressing the imbalance problem through training modification, particularly focusing on gradient descent techniques. We introduce the harmonized gradient descent (HGD) algorithm, which aims to equalize the norms of gradients across different classes. By ensuring the gradient norm balance, HGD mitigates under-fitting for minor classes and achieves balanced online learning. Notably, HGD operates in a streamlined implementation process, requiring no data-buffer, extra parameters, or prior knowledge, making it applicable to any learning models utilizing gradient descent for optimization. Theoretical analysis, based on a few common and mild assumptions, shows that HGD achieves a satisfied sub-linear regret bound. The proposed algorithm are compared with the commonly used online imbalance learning methods under several imbalanced data stream scenarios. Extensive experimental evaluations demonstrate the efficiency and effectiveness of HGD in learning imbalanced data streams.

Problem

• Addresses class imbalance in online data streams
• Equalizes gradient norms across different classes
• Streamlines learning without buffers or extra parameters

Innovation

• Harmonized gradient descent balances class gradient norms
• No data-buffer or extra parameters needed
• Achieves sub-linear regret bound theoretically