Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia

2024-09-13

arXiv.org

AI Summary

This work addresses the limitation of existing LLM evaluations—overemphasis on academic subjects while neglecting real-world professional contexts and regional adaptation—by introducing IndoCareer, the first benchmark for Indonesian localized occupational certification across six sectors: healthcare, finance & insurance, creative design, tourism, education, and law, comprising 8,834 multiple-choice questions. Methodologically, we propose a multi-model zero-/few-shot evaluation framework incorporating answer-order randomization for robustness analysis and cross-domain attribution analysis. Systematic evaluation across 27 mainstream LLMs reveals consistently weak performance on Indonesian professional tasks, with the lowest accuracy observed in finance & insurance; while models exhibit general robustness to answer-order perturbations, significant performance degradation occurs in highly region-specific domains. This study establishes the first systematic LLM assessment of multi-industry occupational competence in Indonesia and identifies critical bottlenecks in modeling regionally grounded professional knowledge.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Application Domains: Humanities & Computational Social Science

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

While knowledge evaluation in large language models has predominantly focused on academic subjects like math and physics, these assessments often fail to capture the practical demands of real-world professions. In this paper, we introduce IndoCareer, a dataset comprising 8,834 multiple-choice questions designed to evaluate performance in vocational and professional certification exams across various fields. With a focus on Indonesia, IndoCareer provides rich local contexts, spanning six key sectors: (1) healthcare, (2) insurance and finance, (3) creative and design, (4) tourism and hospitality, (5) education and training, and (6) law. Our comprehensive evaluation of 27 large language models shows that these models struggle particularly in fields with strong local contexts, such as insurance and finance. Additionally, while using the entire dataset, shuffling answer options generally maintains consistent evaluation results across models, but it introduces instability specifically in the insurance and finance sectors.

Problem

• Evaluating LLMs on real-world professional exams
• Focusing on Indonesia-specific vocational contexts
• Identifying challenges in local context fields

Innovation

• Multi-domain LLM evaluation
• Real-world vocational exams
• Local context dataset IndoCareer