CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants

2026-03-01

AI Summary

Existing benchmarks for large language models fail to capture the practical demands of high-energy physics and high-performance computing, such as scientific constraints, complex dependencies, and performance-critical requirements. To address this gap, this work proposes the first reproducible, automated, and multidimensional evaluation framework tailored to this domain, encompassing three core tasks: Doxygen-style documentation generation, end-to-end GPU kernel synthesis, and vision-enhanced scientific data analysis. By establishing a unified evaluation protocol and an automated scoring mechanism, the framework provides reliable metrics for code correctness, integration capability, robustness, and multimodal validation. This enables fair and systematic comparison of AI programming assistants and establishes a new benchmark for AI-assisted software development in scientific computing.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Other Foundations of Human Computation & AI
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLM) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. The primary contribution of this paper is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks -- code documentation benchmarks measure the ability of an LLM to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development.

Problem

• Large Language Models
• High Energy Physics
• High Performance Computing
• Code Generation
• Benchmarking

Innovation

• HEP/HPC benchmarks
• repeatable evaluation
• scientific coding assistance
• multimodal LLM evaluation
• automated scoring