Ensemble Diversity Optimization for Subjective Supervision

2026-07-09

AI Summary

This work addresses the challenge of modeling predictive uncertainty in subjective NLP tasks, where annotator disagreement introduces systematic divergence that conventional models struggle to capture. The authors propose a novel framework that optimizes ensemble diversity directly in prediction space, jointly learning ensemble weights, effective ensemble size, and calibration within an end-to-end training paradigm. A signed diversity regularizer is introduced to controllably preserve or suppress disagreement, thereby preventing ensemble collapse. The approach integrates Gumbel-Softmax relaxation, a soft F1 surrogate loss, class-weighted cross-entropy, and reliability-weighted diversity regularization. Evaluated on four subjective text classification benchmarks, the model achieves substantially improved probability calibration—reducing cross-entropy by 40–78% and outperforming multiple baselines in Brier score—while maintaining competitive F1 performance and more accurately aligning with the underlying annotator distribution.

Abstract

Subjective NLP tasks often exhibit systematic annotator disagreement, requiring models that represent uncertainty rather than collapse it. We introduce Ensemble Diversity Optimization (EDO), a prediction-space framework that jointly optimizes ensemble weights, effective cardinality, and calibration through a unified differentiable objective. EDO learns ensemble composition and size end-to-end via Gumbel-Softmax relaxation and incorporates a signed diversity regularizer, tuned on validation data, to steer optimization toward either preserving or suppressing disagreement. This regularization prevents ensemble collapse and enables controlled navigation of the utility-calibration trade-off. The framework integrates a soft F1 surrogate, class-weighted cross-entropy to address imbalance, and reliability-weighted diversity to regulate intra-ensemble variability. Experiments on four subjective text-classification benchmarks (ArMIS, ConvAbuse, HS-Brexit, MD-Agreement) show that EDO substantially improves probabilistic calibration, reducing cross-entropy (40-78% depending on baseline) and lowering Brier scores relative to Soft-CE, Soft-MD, Top-5 Voting, and WEL, while maintaining competitive F1 and better alignment with annotator distributions. These results demonstrate that jointly optimizing ensemble structure with a signed diversity regularizer provides an efficient, model-agnostic approach for modeling human subjectivity in supervised learning.

Problem

• subjective NLP
• annotator disagreement
• ensemble diversity
• probabilistic calibration
• uncertainty modeling

Innovation

• Ensemble Diversity Optimization
• subjective supervision
• probabilistic calibration
• diversity regularization
• model-agnostic ensemble