Beyond the Mean: Modelling Annotation Distributions in Continuous Affect Prediction

2026-04-08

AI Summary

This work addresses a key limitation in conventional continuous emotion prediction approaches, which typically compress multiple annotators’ labels into a mean value, thereby discarding valuable information about inter-annotator disagreement and uncertainty. To overcome this, the authors propose a distribution-aware framework that, for the first time in this task, models the full annotation distribution in closed-form. Specifically, emotional consensus is represented using a Beta distribution, and moment matching is employed to map predicted mean and standard deviation to distribution parameters, thereby recovering higher-order statistical properties such as skewness and kurtosis. Integrated with multimodal feature fusion and distributional regression learning, the proposed method not only accurately reconstructs the true annotation distributions on the SEWA and RECOLA datasets but also achieves predictive performance comparable to or better than traditional regression approaches.

Abstract

Emotion annotation is inherently subjective and cognitively demanding, producing signals that reflect diverse perceptions across annotators rather than a single ground truth. In continuous affect prediction, this variability is typically collapsed into point estimates such as the mean or median, discarding valuable information about annotator disagreement and uncertainty. In this work, we propose a distribution-aware framework that models annotation consensus using the Beta distribution. Instead of predicting a single affect value, models estimate the mean and standard deviation of the annotation distribution, which are transformed into valid Beta parameters through moment matching. This formulation enables the recovery of higher-order distributional descriptors, including skewness, kurtosis, and quantiles, in closed form. As a result, the model captures not only the central tendency of emotional perception but also variability, asymmetry, and uncertainty in annotator responses. We evaluate the proposed approach on the SEWA and RECOLA datasets using multimodal features. Experimental results show that Beta-based modelling produces predictive distributions that closely match the empirical annotator distributions while achieving competitive performance with conventional regression approaches. These findings highlight the importance of modelling annotation uncertainty in affective computing and demonstrate the potential of distribution-aware learning for subjective signal analysis.

Problem

• continuous affect prediction
• annotation uncertainty
• subjective annotation
• emotion variability
• distribution modelling

Innovation

• distribution-aware learning
• Beta distribution
• continuous affect prediction
• annotation uncertainty
• moment matching