PBBQ: A Persian Bias Benchmark Dataset Curated with Human-AI Collaboration for Large Language Models

2025-10-22

AI Summary

A lack of culturally grounded evaluation resources hinders the assessment of social biases in large language models (LLMs) within Persian cultural contexts. Method: We introduce PBBQ—the first systematic benchmark for evaluating social bias in Persian LLMs—covering 16 Persian-specific sociocultural dimensions. Co-designed by 250 diverse participants and social science experts, it comprises over 37,000 human-annotated questions. Integrating survey methodology, cultural theory, and AI evaluation techniques, PBBQ employs human-AI collaborative annotation and model-output versus human-response comparative analysis to assess open-source, closed-source, and Persian-finetuned LLMs across multiple bias dimensions. Contribution/Results: Experiments reveal pervasive, statistically significant social biases in current Persian LLMs—strongly aligned with human judgments. The PBBQ dataset will be publicly released, establishing a foundational infrastructure for fairness research in Persian-language AI.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

With the increasing adoption of large language models (LLMs), ensuring their alignment with social norms has become a critical concern. While prior research has examined bias detection in various languages, there remains a significant gap in resources addressing social biases within Persian cultural contexts. In this work, we introduce PBBQ, a comprehensive benchmark dataset designed to evaluate social biases in Persian LLMs. Our benchmark, which encompasses 16 cultural categories, was developed through questionnaires completed by 250 diverse individuals across multiple demographics, in close collaboration with social science experts to ensure its validity. The resulting PBBQ dataset contains over 37,000 carefully curated questions, providing a foundation for the evaluation and mitigation of bias in Persian language models. We benchmark several open-source LLMs, a closed-source model, and Persian-specific fine-tuned models on PBBQ. Our findings reveal that current LLMs exhibit significant social biases across Persian culture. Additionally, by comparing model outputs to human responses, we observe that LLMs often replicate human bias patterns, highlighting the complex interplay between learned representations and cultural stereotypes.Upon acceptance of the paper, our PBBQ dataset will be publicly available for use in future work. Content warning: This paper contains unsafe content.

Problem

• Evaluating social biases in Persian large language models
• Addressing cultural bias gaps in Persian language resources
• Measuring how LLMs replicate human bias patterns in Persian

Innovation

• Human-AI collaboration for bias dataset curation
• Questionnaires from 250 diverse individuals across demographics
• Benchmark dataset with 37000 questions for Persian LLMs