The GaoYao Benchmark: A Comprehensive Framework for Evaluating Multilingual and Multicultural Abilities of Large Language Models

2026-04-22

AI Summary

Existing benchmarks for large language models suffer from fragmented evaluation dimensions, insufficient language coverage, and superficial cultural analysis when assessing multilingual and multicultural capabilities. To address these limitations, this work proposes the GaoYao benchmark, introducing a unified three-layer, nine-dimension framework for cultural cognition evaluation spanning 26 languages and 51 countries or regions. By integrating expert localization, cross-cultural data synthesis, and multi-level cognitive task design, the benchmark constructs a high-quality evaluation suite that extends native-level subjective tasks to 19 languages and enables cross-cultural testing across 34 cultures—achieving up to a 111% improvement in coverage. Systematic evaluation of over 20 mainstream large language models reveals pronounced regional performance disparities and significant capability gaps across tasks, offering a reliable assessment roadmap for global model development.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Evaluating the multilingual and multicultural capabilities of Large Language Models (LLMs) is essential for their global utility. However, current benchmarks face three critical limitations: (1) fragmented evaluation dimensions that often neglect deep cultural nuances; (2) insufficient language coverage in subjective tasks relying on low-quality machine translation; and (3) shallow analysis that lacks diagnostic depth beyond simple rankings. To address these, we introduce GaoYao, a comprehensive benchmark with 182.3k samples, 26 languages and 51 nations/areas. First, GaoYao proposes a unified framework categorizing evaluation tasks into three cultural layers (General Multilingual, Cross-cultural, Monocultural) and nine cognitive sub-layers. Second, we achieve native-quality expansion by leveraging experts to rigorously localize subjective benchmarks into 19 languages and synthesizing cross-cultural test sets for 34 cultures, surpassing prior coverage by up to 111%. Third, we conduct an in-depth diagnostic analysis on 20+ flagship and compact LLMs. Our findings reveal significant geographical performance disparities and distinct gaps between tasks, offering a reliable map for future work. We release the benchmark (https://github.com/lunyiliu/GaoYao).

Problem

• multilingual evaluation
• multicultural capabilities
• benchmark limitations
• cultural nuances
• language coverage

Innovation

• multilingual evaluation
• cultural layers
• native-quality localization
• cross-cultural benchmark
• diagnostic analysis