Automating High Energy Physics Data Analysis with LLM-Powered Agents

2025-12-08

AI Summary

This work addresses the high time and iteration costs of manual coding in high-energy physics (HEP) data analysis. We propose the first LLM-agent-driven automated analysis framework, benchmarked on the Higgs boson diphoton cross-section measurement. Methodologically, we design a supervisor-coder dual-role LLM agent architecture integrated with Snakemake for workflow orchestration, enabling user-instruction-driven code generation, execution, feedback-based correction, and multi-turn iterative refinement. Innovatively, we conduct the first systematic evaluation in HEP of both leading closed-source (Gemini, GPT-4/5, Claude) and open-source LLMs across determinism, stability, and scientific suitability—introducing quantitative metrics including success rate, error taxonomy, and API call overhead. Experiments demonstrate end-to-end automation feasibility, establishing a new paradigm for reproducible scientific computing; baseline code is publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

We present a proof-of-principle study demonstrating the use of large language model (LLM) agents to automate a representative high energy physics (HEP) analysis. Using the Higgs boson diphoton cross-section measurement as a case study with ATLAS Open Data, we design a hybrid system that combines an LLM-based supervisor-coder agent with the Snakemake workflow manager. In this architecture, the workflow manager enforces reproducibility and determinism, while the agent autonomously generates, executes, and iteratively corrects analysis code in response to user instructions. We define quantitative evaluation metrics including success rate, error distribution, costs per specific task, and average number of API calls, to assess agent performance across multi-stage workflows. To characterize variability across architectures, we benchmark a representative selection of state-of-the-art LLMs spanning the Gemini and GPT-5 series, the Claude family, and leading open-weight models. While the workflow manager ensures deterministic execution of all analysis steps, the final outputs still show stochastic variation. Although we set the temperature to zero, other sampling parameters (e.g., top-p, top-k) remained at their defaults, and some reasoning-oriented models internally adjust these settings. Consequently, the models do not produce fully deterministic results. This study establishes the first LLM-agent-driven automated data-analysis framework in HEP, enabling systematic benchmarking of model capabilities, stability, and limitations in real-world scientific computing environments. The baseline code used in this work is available at https://huggingface.co/HWresearch/LLM4HEP. This work was accepted as a poster at the Machine Learning and the Physical Sciences (ML4PS) workshop at NeurIPS 2025. The initial submission was made on August 30, 2025.

Problem

• Automates high energy physics data analysis using LLM agents
• Evaluates LLM performance in generating and correcting analysis code
• Establishes a reproducible framework for benchmarking LLMs in scientific workflows

Innovation

• LLM agents automate HEP analysis code generation and execution
• Hybrid system combines LLM supervisor-coder with Snakemake workflow manager
• Framework enables systematic benchmarking of LLMs in scientific computing