🤖 AI Summary
This work addresses the limitation of existing benchmarks for code-generating large language models, which lack fine-grained assessment of programming proficiency and thus hinder targeted model improvement. To this end, we propose a pedagogy-inspired, multidimensional evaluation framework and introduce CodeEval—a benchmark encompassing 24 Python programming dimensions across three difficulty levels—accompanied by the open-source RunCodeEval execution framework. Our approach uniquely integrates assessment principles from academic programming curricula into large model evaluation, enabling granular diagnostic capabilities through function- and class-level tasks, tiered test cases, and automated metric generation. This system supports fine-grained analysis of model performance across difficulty levels, problem types, and programming constructs. Designed for reproducibility and ease of deployment, the framework empowers researchers to efficiently identify model strengths and weaknesses in specific programming skills, thereby facilitating directed enhancements.
📝 Abstract
Large Language Models (LLMs) are predominantly assessed based on their common sense reasoning, language comprehension, and logical reasoning abilities. While models trained in specialized domains like mathematics or coding have demonstrated remarkable advancements in logical reasoning, there remains a significant gap in evaluating their code generation capabilities. Existing benchmark datasets fall short in pinpointing specific strengths and weaknesses, impeding targeted enhancements in models'reasoning abilities to synthesize code. To bridge this gap, our paper introduces an innovative, pedagogical benchmarking method that mirrors the evaluation processes encountered in academic programming courses. We introduce CodeEval, a multi-dimensional benchmark dataset designed to rigorously evaluate LLMs across 24 distinct aspects of Python programming. The dataset covers three proficiency levels - beginner, intermediate, and advanced - and includes both class-based and function-based problem types with detailed problem specifications and comprehensive test suites. To facilitate widespread adoption, we also developed RunCodeEval, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for CodeEval. RunCodeEval handles test execution, context setup, and metrics generation, enabling researchers to quickly obtain detailed insights into model strengths and weaknesses across complexity levels, problem types, and programming categories. This combination enables targeted evaluation and guides improvements in LLMs'programming proficiencies.