Sports and Women's Sports: Gender Bias in Text Generation with Olympic Data

2025-02-06

AI Summary

This work addresses implicit gender bias in large language models (LLMs) within the sports domain. We construct the first Olympic parallel benchmark featuring matched men’s and women’s events, enabling systematic evaluation of LLMs’ generative biases under gender-ambiguous prompts. Our methodology introduces three quantifiable metrics and a hybrid assessment framework integrating prompt engineering, retrieval-based analysis, and human validation. Results reveal that mainstream LLMs consistently overlook women’s events, preferentially generate men’s competition outcomes, and fail to explicitly annotate gender—indicating systemic erasure of female results in sports contexts. Critically, this bias is both covert and pervasive. Our core contributions are threefold: (1) establishing the first domain-specific bias evaluation benchmark for sports; (2) proposing a reproducible, quantitative methodology for bias measurement; and (3) identifying and empirically validating “unlabeled gender bias”—a novel bias pattern wherein models omit gender specification while implicitly privileging male-associated content.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Large Language Models (LLMs) have been shown to be biased in prior work, as they generate text that is in line with stereotypical views of the world or that is not representative of the viewpoints and values of historically marginalized demographic groups. In this work, we propose using data from parallel men's and women's events at the Olympic Games to investigate different forms of gender bias in language models. We define three metrics to measure bias, and find that models are consistently biased against women when the gender is ambiguous in the prompt. In this case, the model frequently retrieves only the results of the men's event with or without acknowledging them as such, revealing pervasive gender bias in LLMs in the context of athletics.

Problem

• Investigate gender bias in language models
• Measure bias using Olympic event data
• Identify bias against women in ambiguous prompts

Innovation

• Olympic data for bias analysis
• Three metrics measure gender bias
• LLMs show bias in ambiguous prompts