Multi-LCB: Extending LiveCodeBench to Multiple Programming Languages

2026-06-18

AI Summary

Existing code generation benchmarks, such as LiveCodeBench, support only Python, limiting their ability to evaluate the cross-lingual capabilities of large language models. This work extends LiveCodeBench’s tasks to twelve programming languages while preserving its original contamination prevention mechanisms and evaluation protocol, thereby establishing the first fully compatible multilingual benchmark that also supports automatic synchronization with future updates. Through careful task translation, release-date filtering, and contamination control, the benchmark evaluates 24 prominent large language models, revealing significant Python overfitting, language-specific contamination effects, and substantial performance disparities across languages. These findings underscore both the effectiveness and necessity of a truly multilingual evaluation framework for code generation.

Abstract

LiveCodeBench (LCB) has recently become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. By curating competitive programming problems, constantly adding fresh problems to the set, and filtering them by release dates, LCB provides contamination-aware evaluation and offers a holistic view of coding capability. However, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. We introduce Multi-LCB, a benchmark for evaluating LLMs across twelve programming languages, including Python. Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages while preserving LCB's contamination controls and evaluation protocol. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence and requiring models to sustain performance well beyond Python. We evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance. Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.

Problem

• code generation
• large language models
• multi-programming-language evaluation
• benchmark generalization
• Python overfitting

Innovation

• Multi-LCB
• cross-language code generation
• contamination-aware evaluation
• multilingual programming benchmark
• LLM generalization