B-score: Detecting biases in large language models using response history

2025-05-24

AI Summary

This work investigates whether large language models (LLMs) can autonomously correct their own biases by observing multi-turn dialogue histories. To address this, we propose B-score—a generalizable, parameter-free bias quantification metric—designed for subjective, stochastic, and objective question types. B-score operates via response-statistics analysis over dialogue trajectories, requiring no fine-tuning or auxiliary parameters. Experiments demonstrate that LLMs spontaneously debias on stochastic questions; B-score significantly improves bias detection accuracy across MMLU, HLE, and CSQA benchmarks, outperforming verbal confidence estimation and single-turn frequency baselines. Notably, it enables the first unified bias assessment across diverse question dimensions—including subjectivity, stochasticity, and difficulty—thereby facilitating cross-type bias analysis. All code and datasets are publicly released to ensure reproducibility.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) often exhibit strong biases, e.g, against women or in favor of the number 7. We investigate whether LLMs would be able to output less biased answers when allowed to observe their prior answers to the same question in a multi-turn conversation. To understand which types of questions invite more biased answers, we test LLMs on our proposed set of questions that span 9 topics and belong to three types: (1) Subjective; (2) Random; and (3) Objective. Interestingly, LLMs are able to"de-bias"themselves in a multi-turn conversation in response to questions that seek an Random, unbiased answer. Furthermore, we propose B-score, a novel metric that is effective in detecting biases to Subjective, Random, Easy, and Hard questions. On MMLU, HLE, and CSQA, leveraging B-score substantially improves the verification accuracy of LLM answers (i.e, accepting LLM correct answers and rejecting incorrect ones) compared to using verbalized confidence scores or the frequency of single-turn answers alone. Code and data are available at: https://b-score.github.io.

Problem

• Detecting biases in large language models using response history
• Evaluating LLM bias reduction in multi-turn conversations
• Proposing B-score metric for bias detection across question types

Innovation

• Multi-turn conversation reduces LLM biases
• B-score metric detects biases effectively
• Tests span 9 topics, 3 question types