Design-Based Supervised Learning with Noisy Human Labels
This study addresses the bias in classifier training and statistical inference caused by noisy human-reviewed labels. To mitigate this issue, the authors propose Partially Adjudicated Design-based Supervised Learning (PA-DSL), a novel framework that integrates partial expert adjudication into design-based supervised learning. By combining probability sampling audits, label noise correction, and design-weighted estimation, PA-DSL leverages recoverable signals from noisy labels while ensuring unbiased estimation. The method is applicable to various downstream tasks where audit and adjudication probabilities are known. Experiments on synthetic data and semi-synthetic Wikipedia Detox datasets demonstrate that, compared to approaches using only adjudicated labels, PA-DSL reduces root mean squared error by 10%–17% while maintaining nominal coverage.