Learning from Ambiguous Data with Hard Labels

2025-01-03

AI Summary

To address the problem of model overconfidence and degraded generalization caused by label ambiguity in real-world scenarios, this paper challenges the conventional single-hard-label assumption and proposes Quantifiable Label Learning (QLL). QLL models the unobservable soft-label distribution as a quantifiable hard-label sampling process, formally establishing the “ambiguous data + hard labels” learning paradigm for the first time. It introduces a class-perceptive positive–unlabeled (CPU) risk estimator that enables unbiased classifier learning without access to soft labels. Additionally, a hybrid ambiguous-data generation method is developed to enhance robustness. Extensive experiments on multiple benchmark datasets demonstrate that QLL consistently outperforms state-of-the-art hard-label and weakly supervised methods, validating its effectiveness and practicality in improving model generalization.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Real-world data often contains intrinsic ambiguity that the common single-hard-label annotation paradigm ignores. Standard training using ambiguous data with these hard labels may produce overly confident models and thus leading to poor generalization. In this paper, we propose a novel framework called Quantized Label Learning (QLL) to alleviate this issue. First, we formulate QLL as learning from (very) ambiguous data with hard labels: ideally, each ambiguous instance should be associated with a ground-truth soft-label distribution describing its corresponding probabilistic weight in each class, however, this is usually not accessible; in practice, we can only observe a quantized label, i.e., a hard label sampled (quantized) from the corresponding ground-truth soft-label distribution, of each instance, which can be seen as a biased approximation of the ground-truth soft-label. Second, we propose a Class-wise Positive-Unlabeled (CPU) risk estimator that allows us to train accurate classifiers from only ambiguous data with quantized labels. Third, to simulate ambiguous datasets with quantized labels in the real world, we design a mixing-based ambiguous data generation procedure for empirical evaluation. Experiments demonstrate that our CPU method can significantly improve model generalization performance and outperform the baselines.

Problem

• Overconfidence
• Simplified Labels
• Generalization

Innovation

• Quantized Label Learning
• CPU Estimator
• Confidence Calibration