Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypes

2025-08-05

AI Summary

This study investigates whether large language models (LLMs) reproduce psychologically grounded gender stereotypes—such as aggression or gossip propensity—in open-ended narrative generation. Method: We introduce the first multidimensional evaluation framework grounded in 25 empirically validated psychological attributes and three task outcomes, alongside the StereoBias-Stories dataset. Through unconditional and conditional prompting experiments, we systematically analyze bias dynamics across narrative roles and attribute configurations. Contribution/Results: We find that LLMs exhibit significant male-character preference in unconditional generation; irrelevant attributes mitigate bias, whereas stereotype-congruent attribute combinations amplify it; and bias patterns increasingly align with psychological evidence as model scale grows. This work pioneers the integration of structured psychological stereotype constructs into LLM bias assessment, revealing how attribute interactions modulate bias amplification or suppression. It establishes a novel, interpretable paradigm for fairness evaluation in generative AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

As Large Language Models (LLMs) are increasingly used across different applications, concerns about their potential to amplify gender biases in various tasks are rising. Prior research has often probed gender bias using explicit gender cues as counterfactual, or studied them in sentence completion and short question answering tasks. These formats might overlook more implicit forms of bias embedded in generative behavior of longer content. In this work, we investigate gender bias in LLMs using gender stereotypes studied in psychology (e.g., aggressiveness or gossiping) in an open-ended task of narrative generation. We introduce a novel dataset called StereoBias-Stories containing short stories either unconditioned or conditioned on (one, two, or six) random attributes from 25 psychological stereotypes and three task-related story endings. We analyze how the gender contribution in the overall story changes in response to these attributes and present three key findings: (1) While models, on average, are highly biased towards male in unconditioned prompts, conditioning on attributes independent from gender stereotypes mitigates this bias. (2) Combining multiple attributes associated with the same gender stereotype intensifies model behavior, with male ones amplifying bias and female ones alleviating it. (3) Model biases align with psychological ground-truth used for categorization, and alignment strength increases with model size. Together, these insights highlight the importance of psychology-grounded evaluation of LLMs.

Problem

• Investigating gender bias in LLM-generated stories via psychological stereotypes
• Analyzing implicit bias in narrative generation using psychological attributes
• Evaluating gender bias mitigation through stereotype-conditioned story prompts

Innovation

• Using psychological stereotypes for bias analysis
• Introducing StereoBias-Stories dataset for evaluation
• Analyzing gender bias in narrative generation tasks