Conformal-in-the-Loop for Learning with Imbalanced Noisy Data

2024-11-04

arXiv.org

AI Summary

To address the degradation of model performance caused by the coexistence of class imbalance and label noise in real-world scenarios, this paper proposes CitL, a closed-loop conformal learning framework. CitL is the first method to embed conformal prediction into the model training loop, enabling end-to-end differentiable uncertainty quantification for dynamic sample reliability assessment. It achieves adaptive sample reweighting and confidence-driven pruning, jointly mitigating the effects of both label noise and class imbalance—without requiring auxiliary data cleaning or resampling modules. Extensive experiments demonstrate that CitL improves classification accuracy by up to 6.1% and semantic segmentation mIoU by 5.0%, significantly outperforming existing denoising and balancing approaches. The source code is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Computer Vision: Adversarial Attacks & Robustness
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Class imbalance and label noise are pervasive in large-scale datasets, yet much of machine learning research assumes well-labeled, balanced data, which rarely reflects real world conditions. Existing approaches typically address either label noise or class imbalance in isolation, leading to suboptimal results when both issues coexist. In this work, we propose Conformal-in-the-Loop (CitL), a novel training framework that addresses both challenges with a conformal prediction-based approach. CitL evaluates sample uncertainty to adjust weights and prune unreliable examples, enhancing model resilience and accuracy with minimal computational cost. Our extensive experiments include a detailed analysis showing how CitL effectively emphasizes impactful data in noisy, imbalanced datasets. Our results show that CitL consistently boosts model performance, achieving up to a 6.1% increase in classification accuracy and a 5.0 mIoU improvement in segmentation. Our code is publicly available: CitL.

Problem

• Imbalanced Datasets
• Label Noise
• Machine Learning Performance

Innovation

• Conformal-in-the-Loop
• Imbalanced Datasets
• Label Noise