Large Language Models for Code Generation from Multilingual Prompts: A Curated Benchmark and a Study on Code Quality

2026-07-16

AI Summary

This study investigates the impact of multilingual prompts on code generation quality and adherence to programming conventions in large language models, revealing underlying linguistic biases. The authors construct the first high-quality multilingual programming benchmark encompassing Chinese, English, Hindi, Spanish, and Italian, featuring expert human-translated prompts and a multidimensional evaluation framework that includes unit tests, code metrics, static analysis, and lexical features. Using this benchmark, they systematically assess the performance of GPT-4o mini, DeepSeek, and Claude on Python and Java tasks. Their findings indicate that the effect of prompt language on code quality is contingent upon both the target programming language and the model architecture; notably, English prompts do not consistently yield superior results. Moreover, generated code frequently exhibits code-switching, with comments and string literals often mixing the prompt language and English.

Abstract

Large Language Models (LLMs) perform differently on identical programming tasks when prompted in different natural languages, a phenomenon known as language bias. While this behavior has been widely studied for general text generation, its impact on code generation quality and programming conventions remains largely unexplored. We investigate how the language used to describe programming tasks affects the source code generated by GPT-4o mini, DeepSeek, and Claude. Our study comprises 460 coding tasks spanning Python (230) and Java (230). We translate and manually curate the original English prompts into Chinese, Hindi, Spanish, and Italian while preserving their technical meaning. We evaluate the generated code using multiple dimensions, including functional correctness through test pass rates, structural quality using established code metrics, issues detected by static analysis tools, and lexical characteristics such as the language used in identifiers and comments. Our results show that (i) English prompts do not consistently produce the best functional correctness or code quality, (ii) the impact of prompt language depends on both the programming language and the LLM, and (iii) generated code frequently mixes English with the prompt language in comments and string literals. These findings provide the first curated multilingual benchmark for studying language bias in code generation and offer insights for developing more robust multilingual code generation systems.

Problem

• language bias
• code generation
• multilingual prompts
• large language models
• code quality

Innovation

• multilingual code generation
• language bias
• code quality evaluation
• curated benchmark
• large language models