Probing Gender Bias in Multilingual LLMs: A Case Study of Stereotypes in Persian

2025-09-24

AI Summary

This study investigates gender bias in multilingual large language models (LLMs) with respect to Persian—a low-resource language—aiming to mitigate representational harm. We propose the Domain-Specific Gender Shift Index (DS-GSI), the first quantifiable, domain-aware framework for assessing gender bias in Persian. Leveraging template-based probing augmented with authentic corpora, we systematically evaluate bias across four semantic domains—sports, occupation, family, and education—and conduct cross-model comparative analysis. Our empirical findings reveal, for the first time, that mainstream multilingual LLMs exhibit substantial gender bias in Persian, with overall bias levels exceeding those observed in English; sports-related contexts show the most pronounced disparities. This work bridges a critical gap in gender bias research for low-resource languages and establishes a novel methodology and benchmark for cross-lingual fairness evaluation.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Multilingual Large Language Models (LLMs) are increasingly used worldwide, making it essential to ensure they are free from gender bias to prevent representational harm. While prior studies have examined such biases in high-resource languages, low-resource languages remain understudied. In this paper, we propose a template-based probing methodology, validated against real-world data, to uncover gender stereotypes in LLMs. As part of this framework, we introduce the Domain-Specific Gender Skew Index (DS-GSI), a metric that quantifies deviations from gender parity. We evaluate four prominent models, GPT-4o mini, DeepSeek R1, Gemini 2.0 Flash, and Qwen QwQ 32B, across four semantic domains, focusing on Persian, a low-resource language with distinct linguistic features. Our results show that all models exhibit gender stereotypes, with greater disparities in Persian than in English across all domains. Among these, sports reflect the most rigid gender biases. This study underscores the need for inclusive NLP practices and provides a framework for assessing bias in other low-resource languages.

Problem

• Investigating gender bias in multilingual LLMs for Persian language
• Developing methodology to quantify gender stereotypes in low-resource languages
• Comparing bias levels across different semantic domains and languages

Innovation

• Template-based probing methodology for bias detection
• Domain-Specific Gender Skew Index metric introduced
• Evaluated four LLMs focusing on Persian language