Bias-Aware Mislabeling Detection via Decoupled Confident Learning

2025-07-09

AI Summary

This work addresses data integrity issues arising from label bias—specifically, unequal label quality across social groups and the difficulty of detecting and correcting systematic mislabeling. We introduce the first bias-aware mislabeling detection task. To this end, we propose a *Deconfounded Confident Learning* framework that theoretically models confounding bias and disentangles its influence from true class signals, thereby enhancing both detection efficacy and model robustness and fairness. Crucially, our method operates without access to ground-truth labels, relying solely on model prediction confidence scores and group-attribute information to identify bias-affected mislabeled instances. Evaluated on high-stakes real-world tasks—including hate speech detection—our approach significantly outperforms existing mislabeling detection methods, leading to measurable improvements in data quality and downstream model performance.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Reliable data is a cornerstone of modern organizational systems. A notable data integrity challenge stems from label bias, which refers to systematic errors in a label, a covariate that is central to a quantitative analysis, such that its quality differs across social groups. This type of bias has been conceptually and empirically explored and is widely recognized as a pressing issue across critical domains. However, effective methodologies for addressing it remain scarce. In this work, we propose Decoupled Confident Learning (DeCoLe), a principled machine learning based framework specifically designed to detect mislabeled instances in datasets affected by label bias, enabling bias aware mislabelling detection and facilitating data quality improvement. We theoretically justify the effectiveness of DeCoLe and evaluate its performance in the impactful context of hate speech detection, a domain where label bias is a well documented challenge. Empirical results demonstrate that DeCoLe excels at bias aware mislabeling detection, consistently outperforming alternative approaches for label error detection. Our work identifies and addresses the challenge of bias aware mislabeling detection and offers guidance on how DeCoLe can be integrated into organizational data management practices as a powerful tool to enhance data reliability.

Problem

• Detects mislabeled instances in datasets with label bias
• Improves data quality via bias-aware mislabeling detection
• Addresses label bias challenges in hate speech detection

Innovation

• Decoupled Confident Learning for bias detection
• Machine learning framework for mislabeling detection
• Enhances data reliability in hate speech detection