🤖 AI Summary
This study addresses the bias in classifier training and statistical inference caused by noisy human-reviewed labels. To mitigate this issue, the authors propose Partially Adjudicated Design-based Supervised Learning (PA-DSL), a novel framework that integrates partial expert adjudication into design-based supervised learning. By combining probability sampling audits, label noise correction, and design-weighted estimation, PA-DSL leverages recoverable signals from noisy labels while ensuring unbiased estimation. The method is applicable to various downstream tasks where audit and adjudication probabilities are known. Experiments on synthetic data and semi-synthetic Wikipedia Detox datasets demonstrate that, compared to approaches using only adjudicated labels, PA-DSL reduces root mean squared error by 10%–17% while maintaining nominal coverage.
📝 Abstract
Researchers increasingly use automated classifiers to label unstructured data for statistical analysis. Existing rectification methods can correct errors in these automated labels using a probability-sampled audit set, but they usually treat the audit labels as correct. In practice, human audit labels are often noisy, and only some audited items are reviewed by an expert or adjudicator. We propose Partially Adjudicated Design-Based Supervised Learning (PA-DSL), a method for this setting. It uses adjudicated cases to correct noisy human labels and then uses the corrected audit information to debias analyses based on the full set of automated labels. The estimator is valid for a broad class of downstream analyses when the audit and adjudication probabilities are known. In synthetic and Wikipedia Detox semi-synthetic experiments, PA-DSL maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels when noisy human labels contain recoverable signal.