Do Language Models Pass the Bechdel Test? Auditing Gender Biases in LLM-Generated Screenplays

2026-06-22

AI Summary

This study investigates gender representation bias in screenplays generated by large language models (LLMs), with a focus on the underrepresentation of female characters. We introduce an innovative approach that combines an automated Bechdel test with social network analysis—incorporating metrics such as centrality, homophily, and triadic closure—to systematically quantify gender bias in scripts produced by state-of-the-art models including GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5, benchmarking them against human-authored screenplays. Results indicate that human-written scripts are more likely to pass the Bechdel test, yet certain network-based measures reveal that LLM-generated content exhibits less gender bias in specific structural aspects. Nevertheless, all examined scripts—both human- and machine-generated—demonstrate varying degrees of gender representation bias.

Abstract

As large language models (LLMs) are increasingly used in media production from journalistm to filmmaking, what impact do they have on the stories being told? Prior work has shown LLMs to perpetuate social biases, including those related to gender. We complement existing literature on gender bias in LLM outputs by auditing the network structure of LLM-generated movie screenplays through automating the Bechdel test, a popular measure of women's representation in literary and film works. We also introduce the use of social network analysis measures to further analyze representational bias in LLM-generated scripts. We evaluate screenplays generated by three state-of-the-art LLMs (GPT-5, Gemini 3 Pro, and Claude Sonnet 4.5) against 768 corresponding human-written screenplays, finding that human-written scripts are more likely to pass the Bechdel test. However, other network analyses, like centrality, homophily, and triadic relationships demonstrate that in some cases LLM-scripts have less bias, although all script types demonstrate some representational bias under most measures. We conclude by discussing the continued need for further quantitative assessments of media representations and AI-generated content.

Problem

• gender bias
• large language models
• Bechdel test
• screenplay generation
• representational bias

Innovation

• Bechdel test
• social network analysis
• gender bias
• LLM-generated screenplays
• representational bias