Wisdom from Diversity: Bias Mitigation Through Hybrid Human-LLM Crowds

2025-05-18

AI Summary

Large language models (LLMs) often inherit societal biases from training data, exacerbating unfairness—particularly along gender and ethnic dimensions—when responding to sensitive prompts. To address this, we propose the first hybrid crowdsourcing framework that jointly optimizes human diversity and LLM accuracy for bias mitigation. Our method introduces three key innovations: (1) the first systematic demonstration that purely LLM-based crowdsourcing amplifies, rather than reduces, bias; (2) a locally weighted response aggregation mechanism that dynamically calibrates individual response biases; and (3) a human-AI collaborative, cross-subject response integration paradigm. Extensive experiments on multiple ethical evaluation benchmarks show that our approach significantly outperforms state-of-the-art baselines, achieving a new SOTA in bias reduction (average 32.7% decrease) while simultaneously improving the accuracy–fairness trade-off.

Technology Category

• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Despite their performance, large language models (LLMs) can inadvertently perpetuate biases found in the data they are trained on. By analyzing LLM responses to bias-eliciting headlines, we find that these models often mirror human biases. To address this, we explore crowd-based strategies for mitigating bias through response aggregation. We first demonstrate that simply averaging responses from multiple LLMs, intended to leverage the"wisdom of the crowd", can exacerbate existing biases due to the limited diversity within LLM crowds. In contrast, we show that locally weighted aggregation methods more effectively leverage the wisdom of the LLM crowd, achieving both bias mitigation and improved accuracy. Finally, recognizing the complementary strengths of LLMs (accuracy) and humans (diversity), we demonstrate that hybrid crowds containing both significantly enhance performance and further reduce biases across ethnic and gender-related contexts.

Problem

• Mitigating biases in LLMs through hybrid human-LLM crowds
• Addressing bias exacerbation in LLM response aggregation
• Enhancing accuracy and diversity via hybrid crowd strategies

Innovation

• Hybrid human-LLM crowds reduce biases
• Locally weighted aggregation mitigates bias
• Combining human diversity with LLM accuracy