Mechanics of Bias and Reasoning: Interpreting the Impact of Chain-of-Thought Prompting on Gender Bias in LLMs

2026-05-19

AI Summary

This study investigates whether chain-of-thought (CoT) prompting genuinely mitigates gender bias in large language models or merely improves superficial metrics. By integrating gender bias benchmarks, mechanistic interpretability analyses—including attention head clustering and probing of hidden representations—and diagnostic evaluations of CoT reasoning failures, the work systematically demonstrates for the first time that CoT’s debiasing effect is confined to the output layer, while gender bias remains deeply embedded in internal model representations. The findings reveal that CoT’s apparent efficacy primarily stems from memorization of training data rather than genuine semantic reasoning, and it fails to consistently reduce gender bias gaps. This underscores the limitations of current prompting strategies in achieving fundamental debiasing.

Abstract

Large language models (LLMs) are increasingly deployed in socially sensitive settings despite substantial documentation that they encode gender biases. Chain-of-Thought (CoT) prompting has been proposed as a bias-mitigation approach. However, existing evaluations primarily focus on changes in LLM benchmark performance, providing limited insight into whether apparent bias reductions reflect meaningful changes in a model's internal mechanisms. In this work, we investigate how CoT prompting affects gender bias in LLMs, combining benchmark-based evaluation with mechanistic interpretability techniques and reasoning chain failure analysis. Our results confirm a stereotypical bias present in LLM outputs across benchmarks, showing that CoT prompting does not consistently reduce the bias gap. Mechanistic analyses reveal that although CoT balances biased behavior in certain attention head clusters, gender bias remains embedded in hidden representations, indicating only superficial mitigation. Inspection of reasoning chains further suggests that these improvements stem from memorization and familiarity with the dataset rather than genuine understanding of bias.

Problem

• gender bias
• large language models
• Chain-of-Thought prompting
• mechanistic interpretability
• bias mitigation

Innovation

• Chain-of-Thought prompting
• gender bias
• mechanistic interpretability
• attention heads
• reasoning chain analysis