Beyond One-Hot Labels: Semantic Mixing for Model Calibration

2025-04-18

AI Summary

Existing calibration methods rely on deterministic one-hot labels, failing to model epistemic and aleatoric uncertainty realistically, which leads to miscalibrated confidence estimates. To address the scarcity of ground-truth uncertainty annotations, this paper proposes the Calibration-aware Semantic Mixing (CSM) framework: it is the first to jointly leverage the reverse diffusion process and semantic mixing ratio–confidence alignment modeling to synthesize samples with controllable mixing ratios and corresponding ground-truth confidence scores. CSM further introduces a calibration-aware relabeling mechanism and a customized calibration loss. Evaluated across multiple benchmarks, CSM reduces Expected Calibration Error (ECE) by 37% while simultaneously improving both accuracy and robustness. The implementation is publicly available.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Model calibration seeks to ensure that models produce confidence scores that accurately reflect the true likelihood of their predictions being correct. However, existing calibration approaches are fundamentally tied to datasets of one-hot labels implicitly assuming full certainty in all the annotations. Such datasets are effective for classification but provides insufficient knowledge of uncertainty for model calibration, necessitating the curation of datasets with numerically rich ground-truth confidence values. However, due to the scarcity of uncertain visual examples, such samples are not easily available as real datasets. In this paper, we introduce calibration-aware data augmentation to create synthetic datasets of diverse samples and their ground-truth uncertainty. Specifically, we present Calibration-aware Semantic Mixing (CSM), a novel framework that generates training samples with mixed class characteristics and annotates them with distinct confidence scores via diffusion models. Based on this framework, we propose calibrated reannotation to tackle the misalignment between the annotated confidence score and the mixing ratio during the diffusion reverse process. Besides, we explore the loss functions that better fit the new data representation paradigm. Experimental results demonstrate that CSM achieves superior calibration compared to the state-of-the-art calibration approaches. Code is available at github.com/E-Galois/CSM.

Problem

• Model calibration lacks datasets with rich uncertainty annotations
• Existing methods assume perfect certainty in one-hot labels
• Synthetic data generation needed for diverse uncertainty samples

Innovation

• Calibration-aware Semantic Mixing (CSM) framework
• Diffusion models for confidence score annotation
• Calibrated reannotation for score-mixing alignment