MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language

2025-08-01

AI Summary

A severe scarcity of evaluation resources for large language models (LLMs) exists in non-English, non-Western cultural contexts. Method: This study introduces the first systematic evaluation framework tailored to Persian language and Iranian culture, comprising 19 high-quality, domain-specific datasets—covering Iranian law, Persian grammar, idioms, university entrance examinations, and more—curated via rigorous human annotation and culturally aligned validation protocols. We conduct comprehensive benchmarking across 41 mainstream LLMs. Contribution/Results: We release PersianBench, the first large-scale, culturally grounded Persian-language evaluation benchmark. This work establishes foundational standards for cross-cultural LLM assessment, significantly advancing reliability and cultural fidelity evaluation for non-Western language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field.

Problem

• Evaluating LLMs in Persian language lacks benchmarks
• Assessing cultural alignment of LLMs in non-Western contexts
• Bridging gaps in Iranian culture and language evaluation

Innovation

• Introduces 19 Persian-specific evaluation datasets
• Benchmarks 41 LLMs on Iranian cultural topics
• Addresses cultural and linguistic evaluation gaps