L3Cube-IndicQuest v2: A Large-Scale Multilingual Benchmark for Evaluating Factual Knowledge of Large Language Models Across Indic Languages

2026-08-16

AI Summary

This study addresses the absence of evaluation benchmarks for indigenous Indian factual knowledge in large language models by constructing a multilingual dataset comprising 69,000 question-answer pairs across 20 languages. We propose a hybrid construction strategy integrating context-aware generation, semantic deduplication, and human verification to ensure data quality and scalability, while validating the consistency of multiple evaluation protocols. Experimental results demonstrate that proprietary models significantly outperform open-source alternatives, with Gemma-4-31B comprehensively surpassing Sarvam-30B. Furthermore, model rankings remain highly consistent across diverse evaluation protocols. These findings establish a reliable paradigm for assessing regional knowledge capabilities in LLMs, highlighting both current performance disparities and the robustness of the proposed benchmarking framework for low-resource linguistic contexts.

Abstract

We present L3Cube-IndicQuest v2, a large-scale gold-standard multilingual question-answering benchmark for evaluating the India-specific factual knowledge of Large Language Models (LLMs). The benchmark comprises 3,471 curriculum-grounded English question--answer pairs spanning nine domains, curated from educational curricula, competitive examination materials, and domain-specific reference books. We introduce a practical hybrid construction strategy that combines context-grounded LLM-based question generation and validation with semantic deduplication and human verification, enabling scalable creation of benchmark data while preserving annotation quality. The benchmark is translated into 19 Indic languages, yielding a publicly released multilingual dataset of 69,420 question--answer pairs across 20 languages. We evaluate six LLMs under three protocols: LLM-as-a-judge and two deterministic lexical criteria, exact-substring and word-overlap matching. All three produce almost the same model ranking, showing that the results do not depend on the choice of judge. The frontier commercial model leads by a wide margin, and among open-weight models Gemma4 31B outperforms the Indic-specialised Sarvam 30B in every evaluated Indic language.

Problem

• Large Language Models
• Factual Knowledge Evaluation
• Indic Languages
• Multilingual Benchmark

Innovation

• Multilingual Benchmark
• Hybrid Construction Strategy
• Indic Languages
• Factual Knowledge Evaluation
• Evaluation Robustness