TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models

2026-02-26

AI Summary

Existing evaluations of Persian cultural understanding predominantly rely on multiple-choice formats and English-centric metrics, which fail to capture the language’s morphological complexity and semantic nuances. This work proposes the first short-answer question answering benchmark specifically designed for Persian, introducing a rule-based morphological normalization pipeline and a syntax-semantics-integrated soft matching scoring mechanism that overcomes the limitations of exact-match evaluation. Experiments across 15 mainstream large language models demonstrate that the proposed approach improves scoring consistency by 10% over baseline methods, significantly enhancing the accuracy and robustness of cultural understanding assessment. The study also publicly releases the first standardized benchmark for Persian cultural comprehension, establishing a new foundation for future research in this domain.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Machine Learning: Evaluation and Analysis
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Web Mining and Content Analysis: Community question answering

Abstract

This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models, we demonstrate that our hybrid evaluation improves scoring consistency by +10% compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research.

Problem

• cultural competence
• Persian language
• language model evaluation
• short-answer benchmark
• morphological complexity

Innovation

• Persian cultural evaluation
• short-answer benchmark
• morphological normalization
• hybrid syntactic-semantic similarity
• soft-match scoring