AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs Knowledge In STEM Subjects

2024-12-31

AI Summary

This study addresses the lack of evaluation benchmarks for assessing large language models’ (LLMs) comprehension of Arabic scientific knowledge in STEM domains. We introduce ArabSTEM—the first native Arabic, multidisciplinary (mathematics, physics, chemistry, biology), hierarchical, and highly specialized multiple-choice benchmark for STEM reasoning. Curated by domain experts, the dataset undergoes rigorous terminology validation and linguistic localization, supporting zero-shot and few-shot evaluation and integrating seamlessly with mainstream LLM inference frameworks. Experimental results reveal that current multilingual LLMs achieve significantly lower accuracy on ArabSTEM compared to their English counterparts, underscoring substantial challenges in modeling scientific knowledge in Arabic. ArabSTEM fills a critical gap in non-English STEM evaluation infrastructure, providing an essential foundation for developing, benchmarking, and advancing Arabic-capable LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) have shown remarkable capabilities, not only in generating human-like text, but also in acquiring knowledge. This highlights the need to go beyond the typical Natural Language Processing downstream benchmarks and asses the various aspects of LLMs including knowledge and reasoning. Numerous benchmarks have been developed to evaluate LLMs knowledge, but they predominantly focus on the English language. Given that many LLMs are multilingual, relying solely on benchmarking English knowledge is insufficient. To address this issue, we introduce AraSTEM, a new Arabic multiple-choice question dataset aimed at evaluating LLMs knowledge in STEM subjects. The dataset spans a range of topics at different levels which requires models to demonstrate a deep understanding of scientific Arabic in order to achieve high accuracy. Our findings show that publicly available models of varying sizes struggle with this dataset, and underscores the need for more localized language models. The dataset is freely accessible on Hugging Face.

Problem

• Large Language Models
• STEM Knowledge Assessment
• Arabic Scientific Understanding

Innovation

• AraSTEM
• Arabic STEM Assessment
• Language-specific Model Evaluation