AncientBench: Towards Comprehensive Evaluation on Excavated and Transmitted Chinese Corpora

2025-12-19

AI Summary

Existing Chinese benchmarks predominantly focus on modern Chinese and transmitted classical texts, severely lacking coverage of excavated Chinese documents (e.g., bamboo slips, silk manuscripts, bronze inscriptions), thus failing to assess large language models’ understanding of ancient character forms, pronunciations, semantics, and contextual usage. Method: We introduce the first comprehensive ancient Chinese language understanding benchmark integrating both excavated and transmitted texts, grounded in a four-dimensional capability framework—character form, pronunciation, semantics, and context—spanning ten evaluation tasks. We employ collaborative annotation by archaeology experts and incorporate paleographic knowledge to construct high-quality, domain-informed datasets. Contribution/Results: Experiments reveal that state-of-the-art large models significantly underperform human experts on this benchmark. Our work fills a critical gap in AI evaluation for archaeological texts and establishes the first standardized, interdisciplinary evaluation platform for ancient Chinese language understanding and archaeological intelligence.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Comprehension of ancient texts plays an important role in archaeology and understanding of Chinese history and civilization. The rapid development of large language models needs benchmarks that can evaluate their comprehension of ancient characters. Existing Chinese benchmarks are mostly targeted at modern Chinese and transmitted documents in ancient Chinese, but the part of excavated documents in ancient Chinese is not covered. To meet this need, we propose the AncientBench, which aims to evaluate the comprehension of ancient characters, especially in the scenario of excavated documents. The AncientBench is divided into four dimensions, which correspond to the four competencies of ancient character comprehension: glyph comprehension, pronunciation comprehension, meaning comprehension, and contextual comprehension. The benchmark also contains ten tasks, including radical, phonetic radical, homophone, cloze, translation, and more, providing a comprehensive framework for evaluation. We convened archaeological researchers to conduct experimental evaluations, proposed an ancient model as baseline, and conducted extensive experiments on the currently best-performing large language models. The experimental results reveal the great potential of large language models in ancient textual scenarios as well as the gap with humans. Our research aims to promote the development and application of large language models in the field of archaeology and ancient Chinese language.

Problem

• Evaluates large language models' comprehension of ancient Chinese characters
• Focuses on excavated documents not covered by existing benchmarks
• Assesses four competencies: glyph, pronunciation, meaning, and contextual understanding

Innovation

• Proposed AncientBench for excavated document evaluation
• Divided into four comprehension dimensions and ten tasks
• Conducted experiments with archaeological researchers and baseline models