OpenHuEval: Evaluating Large Language Model on Hungarian Specifics

2025-03-27

AI Summary

Existing evaluations of large language models (LLMs) lack rigorous assessment of their performance in Hungarian linguistic and cultural contexts. Method: We introduce HuEval—the first Hungarian-specific benchmark—comprising 3,953 real-user queries across eight language-and-culture dimensions and five task categories. We propose an LLM-as-judge automated evaluation framework, integrating diverse Hungarian corpora, generative-capability-oriented task design, real-query sampling, fine-grained multidimensional metrics, and chain-of-thought analysis using large reasoning models. Contribution/Results: Comprehensive evaluation of mainstream LLMs demonstrates the necessity of Hungarian-specific adaptation. All benchmark data, code, and analytical frameworks are fully open-sourced, significantly enhancing the scientific rigor, precision, and interpretability of non-English LLM evaluation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce OpenHuEval, the first benchmark for LLMs focusing on the Hungarian language and specifics. OpenHuEval is constructed from a vast collection of Hungarian-specific materials sourced from multiple origins. In the construction, we incorporated the latest design principles for evaluating LLMs, such as using real user queries from the internet, emphasizing the assessment of LLMs' generative capabilities, and employing LLM-as-judge to enhance the multidimensionality and accuracy of evaluations. Ultimately, OpenHuEval encompasses eight Hungarian-specific dimensions, featuring five tasks and 3953 questions. Consequently, OpenHuEval provides the comprehensive, in-depth, and scientifically accurate assessment of LLM performance in the context of the Hungarian language and its specifics. We evaluated current mainstream LLMs, including both traditional LLMs and recently developed Large Reasoning Models. The results demonstrate the significant necessity for evaluation and model optimization tailored to the Hungarian language and specifics. We also established the framework for analyzing the thinking processes of LRMs with OpenHuEval, revealing intrinsic patterns and mechanisms of these models in non-English languages, with Hungarian serving as a representative example. We will release OpenHuEval at https://github.com/opendatalab/OpenHuEval .

Problem

• Evaluating LLMs on Hungarian language specifics
• Assessing generative capabilities using real user queries
• Analyzing thinking processes of LRMs in non-English languages

Innovation

• First Hungarian-specific LLM benchmark
• Uses real user queries for evaluation
• Employs LLM-as-judge for accuracy