Mitigating Label Bias with Interpretable Rubric Embeddings

2026-05-20

AI Summary

This work addresses the amplification of historical biases in machine learning models used in high-stakes settings such as hiring and admissions, where training labels often encode systemic inequities. To mitigate this, the authors propose “rubric embeddings”—interpretable representations derived from expert-defined, structured scoring rubrics—that replace conventional black-box features. By anchoring predictions to dimensions semantically aligned with the target construct, this approach disrupts the transmission of bias at the representation level. Integrating domain-driven, interpretable rules into a fairness-aware machine learning framework for the first time, the method significantly reduces inter-group prediction disparities on a large-scale master’s admissions dataset while simultaneously improving the overall quality of admitted applicants, thereby achieving a compelling balance between fairness and predictive validity.

Abstract

Statistical decision algorithms are increasingly deployed in domains where ground-truth labels are hard to obtain, such as hiring, university admissions, and content moderation. In these settings, models are typically trained on historical human evaluations -- for example, using past hiring decisions as a proxy for true applicant quality. However, if past evaluations unjustly favor certain groups, models trained on these labels may inherit those biases. To address this problem, we propose basing predictions on rubric embeddings, a representation framework that replaces standard black-box embeddings with features derived from expert-defined criteria that align with the underlying construct of interest. By anchoring predictions to semantically meaningful dimensions, this approach guards against biased proxy signals. We provide both theoretical and empirical evidence that rubric embeddings mitigate label bias under plausible conditions. Empirically, we evaluate our method on a novel dataset of applications to a large master's program. We find that models trained on rubric embeddings reduce group disparities while improving measures of cohort quality. Our results suggest that basing predictions on interpretable, domain-grounded representations offers a practical approach to learning in the presence of biased labels.

Problem

• label bias
• biased labels
• fairness
• algorithmic decision-making
• proxy labels

Innovation

• rubric embeddings
• label bias mitigation
• interpretable representations
• fair machine learning
• proxy label correction