CS-Bench: A Comprehensive Benchmark for Large Language Models towards Computer Science Mastery

2024-06-12

arXiv.org

AI Summary

Existing LLM evaluations in computer science (CS) overemphasize sub-skills like mathematics and programming, lacking comprehensive, multilingual, and systematic assessment across the full CS domain. Method: We introduce CS-Bench—the first large-scale, multilingual (English, Chinese, French, German) benchmark covering 26 CS subfields with ~10K high-quality, expert-annotated samples. We design a novel multidimensional evaluation framework enabling unified assessment across subfields, task formats (e.g., multiple-choice, open-ended), and languages, grounded in dual-dimension question design (knowledge recall and reasoning) and extensive zero-/few-shot evaluation. Contribution/Results: We systematically evaluate 30+ mainstream LLMs, revealing strong correlations between CS competence and mathematical/programming proficiency, nonlinear scaling with model size, and two critical bottlenecks: domain-specific knowledge gaps and weak reasoning capabilities. We further demonstrate significant cross-task transfer advantages of math/code-specialized models on general CS tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large language models (LLMs) have demonstrated significant potential in advancing various fields of research and society. However, the current community of LLMs overly focuses on benchmarks for analyzing specific foundational skills (e.g. mathematics and code generation), neglecting an all-round evaluation of the computer science field. To bridge this gap, we introduce CS-Bench, the first multilingual (English, Chinese, French, German) benchmark dedicated to evaluating the performance of LLMs in computer science. CS-Bench comprises approximately 10K meticulously curated test samples, covering 26 subfields across 4 key areas of computer science, encompassing various task forms and divisions of knowledge and reasoning. Utilizing CS-Bench, we conduct a comprehensive evaluation of over 30 mainstream LLMs, revealing the relationship between CS performance and model scales. We also quantitatively analyze the reasons for failures in existing LLMs and highlight directions for improvements, including knowledge supplementation and CS-specific reasoning. Further cross-capability experiments show a high correlation between LLMs' capabilities in computer science and their abilities in mathematics and coding. Moreover, expert LLMs specialized in mathematics and coding also demonstrate strong performances in several CS subfields. Looking ahead, we envision CS-Bench serving as a cornerstone for LLM applications in the CS field and paving new avenues in assessing LLMs' diverse reasoning capabilities. The CS-Bench data and evaluation code are available at https://github.com/csbench/csbench.

Problem

• Evaluate LLMs' performance in computer science comprehensively.
• Identify gaps in LLMs' knowledge and reasoning in CS.
• Assess correlation between CS, math, and coding capabilities.

Innovation

• Multilingual benchmark for LLMs in computer science
• 10K test samples across 26 CS subfields
• Evaluation of 30+ LLMs, linking CS performance to scale