Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction

2026-05-13

AI Summary

Existing AI agent evaluation benchmarks struggle to capture the complexity of real-world scientific research tasks, particularly lacking comprehensive assessment of domain knowledge, reasoning, and toolchain integration capabilities essential in high-energy physics analyses. This work proposes the first benchmark platform grounded in publicly available Large Hadron Collider papers and open-source software, requiring AI agents to reproduce published analysis workflows by generating executable simulation and event selection pipelines and predicting signal region event yields. The platform innovatively frames scientific reproduction as a long-horizon tool-use task and introduces a multi-dimensional evaluation framework combining containerized sandbox execution, histogram similarity metrics, computational cost tracking, and a large language model–based judging mechanism to assess both code quality and reasoning trajectories. Experiments reveal that current general-purpose coding agents significantly underperform human physicists on this task, highlighting critical capability gaps in AI-driven scientific replication.

Abstract

Autonomous language-model agents are increasingly evaluated on long-horizon tool-use tasks, but existing benchmarks rarely capture the complexity and nuance of real scientific work. To address this gap, we introduce Collider-Bench, a benchmark for evaluating whether LLM agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software. Such analyses are often difficult to reproduce because the public toolchain only approximates the software used internally by the experimental collaborations, while the published papers inevitably omit implementation details needed for a faithful reconstruction. Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps. Each task requires the agent to turn a published analysis into an executable simulation-and-selection pipeline and submit predicted collision event yields in specified signal regions. These predictions are evaluated with standard histogram metrics that provide continuous fidelity scores without a hand-written rubric. We also report the computational cost incurred by each agent per task. Finally, we evaluate the codebase and full session trace using an LLM judge to catch qualitative failure modes such as fabrications, hallucinations and duplications. We release an initial set of tasks drawn from LHC searches, together with a containerized sandbox and event simulation tools. We evaluate across a capability ladder of general purpose coding agents. Our results show that on average no agent reliably beats the physicist-in-the-loop solution.

Problem

• AI agents
• scientific reproducibility
• particle physics
• LHC analyses
• benchmarking

Innovation

• AI agent benchmarking
• scientific reproducibility
• particle physics analysis
• LLM reasoning
• simulation-based evaluation