Equitable Evaluation via Elicitation

2026-02-24

AI Summary

This study addresses the unfair evaluation of individuals with equivalent qualifications caused by biases stemming from differences in self-presentation styles—such as boasting versus modesty. To mitigate this, the authors propose an interactive AI assessment framework that actively elicits accurate skill disclosures through user-tailored questioning, leveraging synthetic dialogue data generated by large language models for training. The work introduces a novel strict fairness criterion based on covariance constraints, jointly modeling endogenous self-reporting bias and systemic model bias for the first time. By integrating an interactive skill-elicitation mechanism with covariance-regularized learning, the approach significantly weakens the association between evaluation error and expressive style, yielding more accurate and equitable competency assessments in both simulated and real-world settings.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

Individuals with similar qualifications and skills may vary in their demeanor, or outward manner: some tend toward self-promotion while others are modest to the point of omitting crucial information. Comparing the self-descriptions of equally qualified job-seekers with different self-presentation styles is therefore problematic. We build an interactive AI for skill elicitation that provides accurate determination of skills while simultaneously allowing individuals to speak in their own voice. Such a system can be deployed, for example, when a new user joins a professional networking platform, or when matching employees to needs during a company reorganization. To obtain sufficient training data, we train an LLM to act as synthetic humans. Elicitation mitigates endogenous bias arising from individuals' own self-reports. To address systematic model bias we enforce a mathematically rigorous notion of equitability ensuring that the covariance between self-presentation manner and skill evaluation error is small.

Problem

• equitable evaluation
• self-presentation bias
• skill elicitation
• endogenous bias
• evaluation fairness

Innovation

• interactive AI
• skill elicitation
• equitable evaluation
• synthetic data
• endogenous bias