🤖 AI Summary
This work investigates the robustness of Gradient Boosting Decision Trees (GBDTs) to label noise in tabular classification. Addressing the sensitivity of conventional GBDTs to noisy labels and their lack of intrinsic noise detection mechanisms, we propose Gradients—a novel gradient-based noise detection method inspired by deep learning paradigms—and design a GBDT-specific noise-handling pipeline incorporating dynamic relabeling and early stopping. Implemented on XGBoost and LightGBM, our approach jointly leverages prediction confidence, sample consistency, and gradient-derived features for noise identification. Experiments demonstrate state-of-the-art performance: Gradients achieves 99.1% noise detection accuracy on the Adult dataset, substantially outperforming baseline methods. Furthermore, robustness and generalizability are validated across multiple benchmark datasets—including Covertype and Breast Cancer—under diverse noise settings. To our knowledge, this is the first work to adapt deep-learning-inspired noise detection principles to GBDT frameworks, establishing a principled and effective approach for enhancing GBDT reliability in real-world, noisy tabular learning scenarios.
📝 Abstract
Label noise, which refers to the mislabeling of instances in a dataset, can significantly impair classifier performance, increase model complexity, and affect feature selection. While most research has concentrated on deep neural networks for image and text data, this study explores the impact of label noise on gradient-boosted decision trees (GBDTs), the leading algorithm for tabular data. This research fills a gap by examining the robustness of GBDTs to label noise, focusing on adapting two noise detection methods from deep learning for use with GBDTs and introducing a new detection method called Gradients. Additionally, we extend a method initially designed for GBDTs to incorporate relabeling. By using diverse datasets such as Covertype and Breast Cancer, we systematically introduce varying levels of label noise and evaluate the effectiveness of early stopping and noise detection methods in maintaining model performance. Our noise detection methods achieve state-of-the-art results, with a noise detection accuracy above 99% on the Adult dataset across all noise levels. This work enhances the understanding of label noise in GBDTs and provides a foundation for future research in noise detection and correction methods.