Model-agnostic Mitigation Strategies of Data Imbalance for Regression

2025-06-02

AI Summary

Regression tasks involving rare events suffer from severe prediction bias and low reliability due to class imbalance. Method: This paper proposes a model-agnostic framework to mitigate this issue, featuring (i) two novel correlation functions—density-distance and density-ratio—to enhance sample importance interpretability; (ii) two new oversampling techniques, cSMOGN and crbSMOGN; and (iii) a dual-model ensemble strategy balancing performance on both majority and minority samples. The framework integrates density-ratio modeling, enhanced SMOGN, cost-sensitive learning, and multi-model ensembling (neural networks, XGBoost, random forests). Contribution/Results: Evaluated on 10 synthetic and 42 real-world datasets, crbSMOGN achieves statistically significant improvements over state-of-the-art methods when paired with neural networks. The dual-model ensemble effectively prevents performance degradation on majority samples while substantially improving prediction accuracy, calibration, and robustness for rare events.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Data imbalance persists as a pervasive challenge in regression tasks, introducing bias in model performance and undermining predictive reliability. This is particularly detrimental in applications aimed at predicting rare events that fall outside the domain of the bulk of the training data. In this study, we review the current state-of-the-art regarding sampling-based methods and cost-sensitive learning. Additionally, we propose novel approaches to mitigate model bias. To better asses the importance of data, we introduce the density-distance and density-ratio relevance functions, which effectively integrate empirical frequency of data with domain-specific preferences, offering enhanced interpretability for end-users. Furthermore, we present advanced mitigation techniques (cSMOGN and crbSMOGN), which build upon and improve existing sampling methods. In a comprehensive quantitative evaluation, we benchmark state-of-the-art methods on 10 synthetic and 42 real-world datasets, using neural networks, XGBoosting trees and Random Forest models. Our analysis reveals that while most strategies improve performance on rare samples, they often degrade it on frequent ones. We demonstrate that constructing an ensemble of models -- one trained with imbalance mitigation and another without -- can significantly reduce these negative effects. The key findings underscore the superior performance of our novel crbSMOGN sampling technique with the density-ratio relevance function for neural networks, outperforming state-of-the-art methods.

Problem

• Addressing data imbalance bias in regression tasks
• Mitigating model bias for rare event predictions
• Improving predictive reliability with novel sampling techniques

Innovation

• Introduces density-distance and density-ratio relevance functions
• Proposes advanced mitigation techniques cSMOGN and crbSMOGN
• Uses ensemble of models to reduce negative effects