PARAM-1 BharatGen 2.9B Model

2025-07-16

AI Summary

Mainstream large language models (LLMs) exhibit structural linguistic underrepresentation due to their English-centric design, failing to accommodate India’s multilingual reality—characterized by 20+ official languages, pervasive code-switching, and diglossia. Method: We introduce the first natively Indian multilingual LLM, trained from scratch with a decoder-only architecture (2.9B parameters). Our pretraining incorporates language-fair design: a rigid 25% quota of Indian-language data, a morphology-aware, locally adapted SentencePiece tokenizer, and the first India-specific evaluation benchmark. Contribution/Results: The model achieves competitive general-purpose capabilities while substantially outperforming existing SOTA on Indian-domain question answering, mixed-language reasoning, and sociolinguistic robustness tasks. It establishes a new paradigm for equitable multilingual modeling and provides a rigorous, community-grounded baseline for future research.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) have emerged as powerful general-purpose reasoning systems, yet their development remains dominated by English-centric data, architectures, and optimization paradigms. This exclusionary design results in structural under-representation of linguistically diverse regions such as India, where over 20 official languages and 100+ dialects coexist alongside phenomena like code-switching and diglossia. We introduce PARAM-1, a 2.9B parameter decoder-only, text-only language model trained from scratch with an explicit architectural and linguistic focus on Indian diversity. PARAM-1 is trained on a bilingual dataset consisting of only Hindi and English, constructed with a strong focus on fact-rich, high-quality content. It is guided by three core principles: equitable representation of Indic languages through a 25% corpus allocation; tokenization fairness via a SentencePiece tokenizer adapted to Indian morphological structures; and culturally aligned evaluation benchmarks across IndicQA, code-mixed reasoning, and socio-linguistic robustness tasks. By embedding diversity at the pretraining level-rather than deferring it to post-hoc alignment-PARAM-1 offers a design-first blueprint for equitable foundation modeling. Our results demonstrate that it serves as both a competent general-purpose model and a robust baseline for India-centric applications.

Problem

• Addressing under-representation of Indian languages in LLMs
• Developing a Hindi-English bilingual model for India
• Ensuring tokenization fairness and cultural alignment in LLMs

Innovation

• Bilingual Hindi-English dataset for Indian diversity
• SentencePiece tokenizer for Indian morphology
• Culturally aligned evaluation benchmarks