Indi-RomCoM: Code-Mixed Benchmark for Evaluating LLMs on Romanized Indic-English Instructions

2026-06-29

AI Summary

This study addresses the lack of systematic evaluation of large language models (LLMs) on Romanized Hindi–English code-mixed instructions. It introduces the first multidimensional benchmark encompassing four Indian languages, seven task categories, and three levels of code-mixing intensity, evaluated under both zero-shot and few-shot settings across closed-source, open-source, and Indian-language-specialized LLMs. The findings reveal that models generally perform poorly on Romanized code-mixed instructions, with performance degrading significantly as mixing density increases. Notably, reasoning tasks exhibit less performance degradation than detection tasks, likely because generated explanations provide additional contextual cues. This benchmark establishes a new evaluation paradigm for assessing LLMs in low-resource code-mixed scenarios.

Abstract

Romanized Code Mixing (RCM), where bilingual speakers fluidly blend local languages with English in Roman script, has emerged as the dominant form of communication across multilingual communities. While Large Language Models (LLMs) perform strongly on monolingual and native-script benchmarks, their ability to follow instructions and reason over RCM-based content remains largely unexplored. To this end, we introduce the Indi-RomCoM benchmark for facilitating systematic evaluation on Indic Romanized Code-Mixed instructions. Our benchmark spans seven instruction-following tasks, four widely spoken Indic languages, and three controlled code-mixing intensity levels. We extensively evaluate a suite of LLMs covering proprietary, open-weight, and Indic-focused models under zero- and few-shot settings. LLMs consistently underperform on RCM instructions, with performance degrading as code-mixing density increases. Furthermore, reasoning tasks suffer less degradation than detection tasks (e.g., Toxicity) because the generated explanations offer necessary context. We believe Indi-RomCoM helps the community in developing inclusive multilingual systems.

Problem

• Romanized Code Mixing
• Indic-English
• Instruction Following
• Multilingual Evaluation
• Large Language Models

Innovation

• Romanized Code Mixing
• Indic-English
• instruction-following benchmark
• multilingual LLM evaluation
• code-mixing intensity