Partial-Label Learning with a Reject Option

2024-02-01

AI Summary

To address high-risk decisions arising from ambiguous annotations in partial-label learning (PLL), this paper proposes Risk-consistent Nearest Neighbor (RiNN), the first PLL framework incorporating a theoretically risk-consistent rejection mechanism. RiNN integrates nearest-neighbor-based weak supervision modeling, a risk-consistent loss function, and a confidence-adaptive rejection strategy—bypassing the suboptimal trade-off between rejection rate and accuracy inherent in threshold-based methods. Empirical evaluation across multiple synthetic and real-world datasets demonstrates that RiNN significantly improves the precision–recall balance under rejection settings; even without rejection, it achieves state-of-the-art predictive performance among PLL methods. The core contribution lies in a theory-driven, risk-controllable rejection mechanism and its effective instantiation within weakly supervised learning.

Technology Category

• Machine Learning: Active Learning
• Reasoning under Uncertainty: Sequential Decision Making
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

In real-world applications, one often encounters ambiguously labeled data, where different annotators assign conflicting class labels. Partial-label learning allows training classifiers in this weakly supervised setting, where state-of-the-art methods already show good predictive performance. However, even the best algorithms give incorrect predictions, which can have severe consequences when they impact actions or decisions. We propose a novel risk-consistent nearest-neighbor-based partial-label learning algorithm with a reject option, that is, the algorithm can reject unsure predictions. Extensive experiments on artificial and real-world datasets show that our method provides the best trade-off between the number and accuracy of non-rejected predictions when compared to our competitors, which use confidence thresholds for rejecting unsure predictions. When evaluated without the reject option, our nearest-neighbor-based approach also achieves competitive prediction performance.

Problem

• Partial Label Learning
• Prediction Accuracy
• Uncertainty Control

Innovation

• Selective Prediction
• Partial Label Learning
• Uncertainty Control