CoCo-Bench: A Comprehensive Code Benchmark For Multi-task Large Language Model Evaluation

2025-04-29

AI Summary

Existing code benchmarks predominantly target single tasks, failing to comprehensively evaluate large language models’ holistic capabilities in realistic software engineering scenarios. To address this gap, we propose CodeBench—the first multilingual, multi-difficulty benchmark systematically covering four core dimensions: code understanding, generation, modification, and review. Our methodology features: (1) the first principled integration of essential developer competencies; (2) high-quality, human-annotated data construction; and (3) a cross-task consistent evaluation framework. Extensive experiments demonstrate that CodeBench precisely identifies capability bottlenecks across state-of-the-art code LMs, significantly discriminates model performance across diverse subtasks, and exhibits strong reliability and scalability. As a result, CodeBench establishes a reproducible, extensible, and empirically grounded standard for comprehensive code model evaluation.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large language models (LLMs) play a crucial role in software engineering, excelling in tasks like code generation and maintenance. However, existing benchmarks are often narrow in scope, focusing on a specific task and lack a comprehensive evaluation framework that reflects real-world applications. To address these gaps, we introduce CoCo-Bench (Comprehensive Code Benchmark), designed to evaluate LLMs across four critical dimensions: code understanding, code generation, code modification, and code review. These dimensions capture essential developer needs, ensuring a more systematic and representative evaluation. CoCo-Bench includes multiple programming languages and varying task difficulties, with rigorous manual review to ensure data quality and accuracy. Empirical results show that CoCo-Bench aligns with existing benchmarks while uncovering significant variations in model performance, effectively highlighting strengths and weaknesses. By offering a holistic and objective evaluation, CoCo-Bench provides valuable insights to guide future research and technological advancements in code-oriented LLMs, establishing a reliable benchmark for the field.

Problem

• Existing benchmarks lack comprehensive evaluation for real-world code tasks
• No unified framework assesses code understanding, generation, modification, and review
• Current benchmarks are limited in programming languages and task diversity

Innovation

• Multi-dimensional evaluation framework for LLMs
• Includes multiple programming languages and difficulties
• Rigorous manual review ensures data quality