CorIL: Towards Enriching Indian Language to Indian Language Parallel Corpora and Machine Translation Systems

2025-09-24

AI Summary

High-quality, cross-domain parallel corpora for Indian languages are severely scarce, hindering multilingual machine translation (MT) research and deployment. To address this, we introduce IndicParallel—the first large-scale, domain-annotated (government, health, general) parallel corpus covering 11 official Indian languages, comprising 772K sentence pairs. All data undergo rigorous human verification and systematic cleaning to ensure high fidelity. We establish a unified benchmark on leading neural MT models—including IndicTrans2, NLLB, and BhashaVerse—enabling reproducible evaluation. Our empirical analysis is the first to quantitatively reveal a significant performance gap between Persian-Arabic scripts (e.g., Urdu) and Indic scripts (e.g., Devanagari) in multilingual modeling, underscoring both the corpus’s inherent difficulty and its practical utility for domain-aware translation and cross-script transfer studies.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

India's linguistic landscape is one of the most diverse in the world, comprising over 120 major languages and approximately 1,600 additional languages, with 22 officially recognized as scheduled languages in the Indian Constitution. Despite recent progress in multilingual neural machine translation (NMT), high-quality parallel corpora for Indian languages remain scarce, especially across varied domains. In this paper, we introduce a large-scale, high-quality annotated parallel corpus covering 11 of these languages : English, Telugu, Hindi, Punjabi, Odia, Kashmiri, Sindhi, Dogri, Kannada, Urdu, and Gujarati comprising a total of 772,000 bi-text sentence pairs. The dataset is carefully curated and systematically categorized into three key domains: Government, Health, and General, to enable domain-aware machine translation research and facilitate effective domain adaptation. To demonstrate the utility of CorIL and establish strong benchmarks for future research, we fine-tune and evaluate several state-of-the-art NMT models, including IndicTrans2, NLLB, and BhashaVerse. Our analysis reveals important performance trends and highlights the corpus's value in probing model capabilities. For instance, the results show distinct performance patterns based on language script, with massively multilingual models showing an advantage on Perso-Arabic scripts (Urdu, Sindhi) while other models excel on Indic scripts. This paper provides a detailed domain-wise performance analysis, offering insights into domain sensitivity and cross-script transfer learning. By publicly releasing CorIL, we aim to significantly improve the availability of high-quality training data for Indian languages and provide a valuable resource for the machine translation research community.

Problem

• Addressing scarce high-quality parallel corpora for Indian languages
• Enabling domain-aware machine translation across diverse Indian languages
• Establishing benchmarks for cross-script transfer learning analysis

Innovation

• Created large-scale parallel corpus for 11 Indian languages
• Categorized data into Government, Health, and General domains
• Fine-tuned state-of-the-art NMT models for benchmarking