Label Distribution Learning using the Squared Neural Family on the Probability Simplex

2024-12-10

arXiv.org

AI Summary

Existing label distribution learning (LDL) methods primarily focus on point estimation, failing to characterize the uncertainty of label distributions over the probability simplex. This work proposes the first probabilistic modeling framework explicitly targeting the label distribution itself—instead of predicting a single optimal distribution, it directly models the posterior distribution of label distributions across the entire simplex. Innovatively, we introduce the Squared Neural Family (SNEFY) onto the probability simplex, enabling differentiable, normalized distribution-family modeling. The framework supports expectation-based prediction, uncertainty quantification, and reliability calibration. Evaluated on standard LDL benchmarks, our method achieves state-of-the-art performance. Moreover, it significantly enhances active learning and ensemble learning efficacy, empirically validating the effectiveness and robustness of uncertainty-aware prediction.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Label distribution learning (LDL) provides a framework wherein a distribution over categories rather than a single category is predicted, with the aim of addressing ambiguity in labeled data. Existing research on LDL mainly focuses on the task of point estimation, i.e., pinpointing an optimal distribution in the probability simplex conditioned on the input sample. In this paper, we estimate a probability distribution of all possible label distributions over the simplex, by unleashing the expressive power of the recently introduced Squared Neural Family (SNEFY). With the modeled distribution, label distribution prediction can be achieved by performing the expectation operation to estimate the mean of the distribution of label distributions. Moreover, more information about the label distribution can be inferred, such as the prediction reliability and uncertainties. We conduct extensive experiments on the label distribution prediction task, showing that our distribution modeling based method can achieve very competitive label distribution prediction performance compared with the state-of-the-art baselines. Additional experiments on active learning and ensemble learning demonstrate that our probabilistic approach can effectively boost the performance in these settings, by accurately estimating the prediction reliability and uncertainties.

Problem

• Estimates probability distribution of label distributions over simplex
• Derives closed-form label distribution statistics for uncertainty quantification
• Enhances LDL with confidence intervals and correlation measurements

Innovation

• Uses Squared Neural Family for probability modeling
• Estimates full label distribution over simplex
• Derives closed-form statistics for uncertainty quantification