Learning Confidence Bounds for Classification with Imbalanced Data

2024-07-16

European Conference on Artificial Intelligence

AI Summary

Class imbalance in classification tasks often induces model bias and unreliable predictions. To address this, we propose a novel learning paradigm that neither relies on resampling nor loss reweighting: it incorporates class-dependent confidence bounds as learnable modules within the classification framework. Grounded in Hoeffding-type concentration inequalities and class-conditional risk minimization, our approach adaptively models both class-wise uncertainty and imbalance severity. Derived from statistical learning theory, it avoids the information loss and estimation bias inherent in conventional undersampling or oversampling strategies. Evaluated on multiple benchmark imbalanced datasets, our method significantly improves minority-class recall while reducing calibration error by up to 32%. Moreover, predicted confidence scores exhibit strong alignment with empirical accuracy, substantially enhancing prediction robustness and reliability.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Stochastic Optimization
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Class imbalance poses a significant challenge in classification tasks, where traditional approaches often lead to biased models and unreliable predictions. Undersampling and oversampling techniques have been commonly employed to address this issue, yet they suffer from inherent limitations stemming from their simplistic approach such as loss of information and additional biases respectively. In this paper, we propose a novel framework that leverages learning theory and concentration inequalities to overcome the shortcomings of traditional solutions. We focus on understanding the uncertainty in a class-dependent manner, as captured by confidence bounds that we directly embed into the learning process. By incorporating class-dependent estimates, our method can effectively adapt to the varying degrees of imbalance across different classes, resulting in more robust and reliable classification outcomes. We empirically show how our framework provides a promising direction for handling imbalanced data in classification tasks, offering practitioners a valuable tool for building more accurate and trustworthy models.

Problem

• Addressing class imbalance in classification tasks
• Overcoming limitations of traditional sampling techniques
• Incorporating class-dependent confidence bounds for robustness

Innovation

• Learning theory and concentration inequalities framework
• Class-dependent confidence bounds embedded in learning
• Adapting to varying imbalance degrees across classes