Do They Understand Them? An Updated Evaluation on Nonbinary Pronoun Handling in Large Language Models

2025-08-01

AI Summary

This study addresses the accuracy and fairness deficiencies of large language models (LLMs) in processing non-binary pronouns—including gender-neutral pronouns and neopronouns—by proposing and constructing an updated benchmark, MISGENDERED+. The benchmark expands task dimensions (zero-shot and few-shot prompting, forward and reverse gender-identity inference) and broadens model coverage to include GPT-4o, Claude 3, DeepSeek-V3, Qwen Turbo, and Qwen2.5. It constitutes the first systematic evaluation of LLMs’ neopronoun recognition capability and robustness in reverse identity inference. Experimental results show marked performance improvements on binary and common gender-neutral pronouns across mainstream models; however, significant biases persist in neopronoun identification and reverse inference tasks, exposing fundamental limitations in identity-sensitive reasoning. This work establishes a more comprehensive and challenging evaluation framework for assessing the gender inclusivity of LLMs, advancing rigorous, equitable assessment methodologies in responsible AI development.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Large language models (LLMs) are increasingly deployed in sensitive contexts where fairness and inclusivity are critical. Pronoun usage, especially concerning gender-neutral and neopronouns, remains a key challenge for responsible AI. Prior work, such as the MISGENDERED benchmark, revealed significant limitations in earlier LLMs' handling of inclusive pronouns, but was constrained to outdated models and limited evaluations. In this study, we introduce MISGENDERED+, an extended and updated benchmark for evaluating LLMs' pronoun fidelity. We benchmark five representative LLMs, GPT-4o, Claude 4, DeepSeek-V3, Qwen Turbo, and Qwen2.5, across zero-shot, few-shot, and gender identity inference. Our results show notable improvements compared with previous studies, especially in binary and gender-neutral pronoun accuracy. However, accuracy on neopronouns and reverse inference tasks remains inconsistent, underscoring persistent gaps in identity-sensitive reasoning. We discuss implications, model-specific observations, and avenues for future inclusive AI research.

Problem

• Evaluating LLMs' accuracy with gender-neutral pronouns
• Assessing neopronoun handling in modern language models
• Identifying gaps in identity-sensitive AI reasoning

Innovation

• Extended benchmark MISGENDERED+ for pronoun evaluation
• Tested five LLMs across multiple inference scenarios
• Assessed binary, gender-neutral, and neopronoun accuracy