Beyond Bilingual Transfer: Multilingual Code-Switching in Instruction Tuning

2026-05-28

AI Summary

This study addresses the limited scope of existing research, which predominantly focuses on bilingual transfer between English and a single target language, by systematically exploring code-mixed scenarios involving three or more languages. We construct the first sentence-level multilingual code-mixed instruction dataset encompassing English, Japanese, Korean, and Chinese, and fine-tune multilingual large language models using this data. The resulting models are evaluated on the Belebele benchmark to assess their multilingual comprehension capabilities. Experimental results demonstrate that our approach consistently improves average performance across all four languages, effectively overcoming the constraints of traditional bilingual transfer and significantly expanding the applicability of code-mixed data in multilingual learning.

Abstract

Recent studies have shown that code-switching data (CSD), in which multiple languages are mixed within the same context, can improve cross-lingual transfer and multilingual alignment in large language models (LLMs). However, existing studies primarily focus on bilingual transfer between English and a target language, leaving multilingual settings involving three or more languages largely unexplored. In this work, we investigate multilingual code-switching instruction tuning across four languages: English, Japanese, Korean, and Chinese. We evaluate multilingual understanding on Belebele. Our experiments show that simple sentence-level multilingual CSD consistently improves average multilingual performance across all four languages, indicating that multilingual code-switching can be effective beyond bilingual transfer settings.

Problem

• multilingual code-switching
• instruction tuning
• cross-lingual transfer
• large language models

Innovation

• multilingual code-switching
• instruction tuning
• cross-lingual transfer
• large language models
• multilingual alignment