Beyond Pass Rate: A Multilingual, Execution-Grounded Evaluation of Open Code LLMs

2026-06-07

AI Summary

Current evaluations of large code models predominantly rely on a single pass rate metric, which fails to capture their true performance across multiple programming languages, problem types, and error categories. This work proposes a multilingual, fine-grained evaluation framework that integrates both execution-based testing and static code analysis. We conduct a large-scale assessment of nine open-source models across 12 programming languages and 2,707 LeetCode problems. Results reveal that even the best-performing model, Yi-Coder-9B-Chat, achieves only a 23.64% average accuracy—substantially lower than the human baseline of 57.2%. Notably, 63.25% of failures stem from compilation errors, and static code quality shows a significant disconnect from functional correctness, exposing critical performance limitations obscured by conventional single-metric evaluations.

Abstract

Code generation models are typically compared using compact execution benchmarks and aggregate pass rates, but such summaries obscure how performance varies across programming languages, problem families, and failure modes. We present a large-scale, execution-grounded evaluation of 9 openly accessible LLMs specialized for coding on 2,707 free LeetCode problems across 12 programming languages. Our corpus contains 325,343 problem-model-language jobs, each linked to prompt metadata, extracted code, LeetCode execution outcomes, and static-analysis signals. The results show that current open models remain far from the human acceptance reference: the best model, Yi-Coder-9B-Chat, reaches 23.64% mean correctness, compared with a 57.2% human acceptance baseline. Rankings are also slice-dependent: Qwen2.5-Coder-14B-Instruct is strongest on hard problems and distinct-problem coverage, while Gemma-2-27B-IT achieves the highest all-language lint pass rate. Failure analysis shows that compile errors account for 63.25% of non-accepted best submissions, indicating that many failures occur before semantic correctness can be tested. Static quality further diverges from functional correctness. Together, these findings show that multilingual, artifact-preserving evaluation reveals tradeoffs hidden by single-language or single-metric leaderboards.

Problem

• code generation
• large language models
• multilingual evaluation
• execution-based benchmarking
• model evaluation

Innovation

• multilingual evaluation
• execution-grounded benchmarking
• code LLMs
• static analysis
• failure mode analysis