PROBE: Benchmarking Code Generation in Large Language Models

2026-07-15

AI Summary

Current evaluations of code generation predominantly rely on single-language benchmarks and unit tests, which inadequately capture the full spectrum of model capabilities. This work proposes PROBE—a scalable, multidimensional evaluation framework that systematically integrates five programming languages, multiple prompting strategies, and diverse metrics to holistically assess six prominent large language models across three key dimensions: functional correctness, solution proximity, and code quality. Leveraging functional testing, semantic similarity analysis, and static code quality assessment, the study reveals that while models perform well on simple tasks and in high-resource languages, they often produce unreliable outputs on complex problems or in low-resource languages due to fundamental errors, thereby exposing critical limitations in current code generation capabilities.

Abstract

Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks, particularly in automated code generation. Despite their widespread adoption, these models remain far from perfect, making systematic and fair evaluation essential to understand their strengths and limitations. In the context of code generation, existing benchmarks are limited: they often target a single programming language and rely primarily on unit test outcomes, while overlooking other critical dimensions such as the overall quality of the generated code and its closeness to a valid solution. To address these gaps, we introduce PROBE, an extensible benchmark framework that, unlike prior work, establishes a systematic structure built on diverse and well-defined metrics, representative workloads, varied prompt templates, and a robust experimental procedure. In practice, the code generated by the LLMs is evaluated along three complementary dimensions: functional correctness, proximity to valid solutions, and code quality, enabling a comprehensive assessment of performance. We use PROBE to evaluate four open-source and two proprietary models under three prompting strategies across five programming languages. We further complement this analysis with a study of common errors in the code and provide concrete examples, offering clearer insight into where LLMs tend to struggle. Our findings show that, while LLMs achieve promising results, they struggle with harder problems and, in the case of smaller models, with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.

Problem

• code generation
• large language models
• benchmarking
• evaluation metrics
• programming languages

Innovation

• code generation
• benchmarking
• large language models
• code quality
• functional correctness