An Assessment of Human vs. Model Uncertainty in Soft-Label Learning and Calibration

2026-05-18

AI Summary

This study investigates whether the improved calibration from human soft labels stems from their expression of uncertainty or from implicit correction of erroneous hard labels—i.e., distributional shift. By relabeling a subset of MNIST and its synthetic variants to capture human uncertainty, the work disentangles the effects of soft-label supervision and label correction while controlling for distributional shift. The authors introduce a diagnostic framework for aligning human and model uncertainty, integrating soft-label learning, dataset cartography, and calibration analysis. They find that although human soft labels yield only modest gains in accuracy, they act as an effective regularizer, significantly enhancing calibration on difficult examples and improving training stability, thereby aligning model uncertainty more closely with human judgment.

Abstract

Central to human-aligned AI is understanding the benefits of human-elicited labels over synthetic alternatives. While human soft-labels improve calibration by capturing uncertainty, prior studies conflate these benefits with the implicit correction of mislabeled data (mode shifts), obscuring true effects of soft-labels. We present a controlled audit of soft-label learning across MNIST and a synthetic variant, re-annotating subsets to extract human uncertainty. By decoupling soft-label supervision from underlying label mode shifts, we show that while human soft-labels do provide accuracy gains, their larger value lies in acting as a regularizer that improves model calibration on difficult samples and promotes stable convergence across training runs. Dataset cartography reveals models trained on human soft-labels mirror human uncertainty, whereas those trained on synthetic labels fail to align with humans. Broadly, this work provides a diagnostic testbed for human-AI uncertainty alignment.

Problem

• soft-label learning
• model calibration
• human uncertainty
• label mode shift
• human-AI alignment

Innovation

• soft-label learning
• model calibration
• human uncertainty
• label mode shift
• dataset cartography