ELAB: Extensive LLM Alignment Benchmark in Persian Language

2025-04-17

AI Summary

This work addresses the lack of culturally grounded evaluation frameworks for Persian large language models (LLMs) across critical ethical dimensions—safety, fairness, and social norm alignment. Methodologically, we propose a multi-source data construction paradigm integrating machine translation, LLM-based synthesis, and human curation; design a cross-dimensional evaluation framework; and introduce culture-sensitive prompt engineering alongside standardized assessment protocols. Our key contributions include: (1) releasing the first culturally aware Persian alignment benchmark suite (e.g., ProhibiBench-fa, GuardBench-fa); (2) establishing an open-source, unified evaluation platform with a public leaderboard. Comprehensive experiments reveal substantial capability gaps among state-of-the-art Persian LLMs across all three ethical dimensions, thereby filling a critical localization gap in alignment evaluation for non-English LLMs.

Technology Category

• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Large Multimodal Models (LMMs)
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

This paper presents a comprehensive evaluation framework for aligning Persian Large Language Models (LLMs) with critical ethical dimensions, including safety, fairness, and social norms. It addresses the gaps in existing LLM evaluation frameworks by adapting them to Persian linguistic and cultural contexts. This benchmark creates three types of Persian-language benchmarks: (i) translated data, (ii) new data generated synthetically, and (iii) new naturally collected data. We translate Anthropic Red Teaming data, AdvBench, HarmBench, and DecodingTrust into Persian. Furthermore, we create ProhibiBench-fa, SafeBench-fa, FairBench-fa, and SocialBench-fa as new datasets to address harmful and prohibited content in indigenous culture. Moreover, we collect extensive dataset as GuardBench-fa to consider Persian cultural norms. By combining these datasets, our work establishes a unified framework for evaluating Persian LLMs, offering a new approach to culturally grounded alignment evaluation. A systematic evaluation of Persian LLMs is performed across the three alignment aspects: safety (avoiding harmful content), fairness (mitigating biases), and social norms (adhering to culturally accepted behaviors). We present a publicly available leaderboard that benchmarks Persian LLMs with respect to safety, fairness, and social norms at: https://huggingface.co/spaces/MCILAB/LLM_Alignment_Evaluation.

Problem

• Evaluating Persian LLMs on ethical alignment
• Addressing gaps in Persian cultural and linguistic contexts
• Creating comprehensive Persian benchmarks for LLM evaluation

Innovation

• Adapts LLM evaluation frameworks to Persian context
• Creates Persian benchmarks with translated and new data
• Evaluates safety, fairness, and social norms systematically