Quantifying Ambiguity in Categorical Annotations: A Measure and Statistical Inference Framework
This paper addresses soft label distributions arising from semantic ambiguity—not annotation errors—in classification tasks. We propose a novel method to quantify aleatoric uncertainty by introducing an asymmetric “undecidable” class that distinguishes between inter-class indistinguishability and intrinsic ambiguity. Our approach defines a fuzziness measure based on a refined quadratic entropy (Gini impurity) and integrates it into a Bayesian inference framework with a Dirichlet prior, jointly modeling epistemic and aleatoric uncertainty. The framework supports both frequentist point estimation and Bayesian posterior inference. Crucially, it maps discrete response distributions to scalar interpretability metrics in [0,1], enabling quantifiable, group-level ambiguity assessment. Experiments demonstrate superior performance in uncertainty calibration and data quality evaluation, effectively informing downstream machine learning pipeline optimization.