Adapting Psycholinguistic Research for LLMs: Gender-inclusive Language in a Coreference Context

2025-02-18

AI Summary

This study investigates whether large language models (LLMs) exhibit gender neutrality in coreference resolution involving gender-inclusive language (e.g., non-binary or gender-neutral expressions) and uncovers latent cross-linguistic gender biases. Methodologically, it innovatively adapts a French psycholinguistic paradigm—first extended to English and German—combining prompt engineering with controlled cloze tasks across Llama, GPT, and Claude models, validated via statistical significance testing. Results reveal that while English LLMs generally preserve antecedent gender, they exhibit an underlying male bias; in German, this bias is markedly stronger, systematically overriding diverse gender-neutral strategies. Crucially, the study demonstrates how grammatical gender systems amplify implicit biases in LLMs—a previously undocumented phenomenon. It thus establishes a novel methodology and empirical benchmark for cross-linguistic fairness evaluation in NLP.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Gender-inclusive language is often used with the aim of ensuring that all individuals, regardless of gender, can be associated with certain concepts. While psycholinguistic studies have examined its effects in relation to human cognition, it remains unclear how Large Language Models (LLMs) process gender-inclusive language. Given that commercial LLMs are gaining an increasingly strong foothold in everyday applications, it is crucial to examine whether LLMs in fact interpret gender-inclusive language neutrally, because the language they generate has the potential to influence the language of their users. This study examines whether LLM-generated coreferent terms align with a given gender expression or reflect model biases. Adapting psycholinguistic methods from French to English and German, we find that in English, LLMs generally maintain the antecedent's gender but exhibit underlying masculine bias. In German, this bias is much stronger, overriding all tested gender-neutralization strategies.

Problem

• Examines LLMs' processing of gender-inclusive language
• Investigates LLMs' biases in gender coreference resolution
• Adapts psycholinguistic methods to assess LLM performance

Innovation

• Adapts psycholinguistic methods for LLMs
• Examines gender-inclusive language in LLMs
• Assesses coreferent terms for gender bias