Agent-based code generation for the Gammapy framework

2025-09-30

AI Summary

Scientific libraries such as Gammapy suffer from sparse documentation, unstable APIs, and limited community support, severely hindering large language models (LLMs) from generating reliable code. Method: We propose an autonomous code generation agent tailored for scientific frameworks, integrating an LLM, an executable code sandbox, and a lightweight web interface to establish a “generate–execute–verify” closed-loop workflow—reducing reliance on comprehensive documentation and API stability. Contribution/Results: Our approach introduces a dynamic feedback-driven iterative debugging mechanism and a verification protocol specifically designed for scientific computing. We implement a prototype system and a benchmarking suite, empirically validating its effectiveness and scalability on real-world gamma-ray data analysis tasks. Results demonstrate substantial improvements in the robustness and practical utility of LLM-generated code within resource-constrained, domain-specific scientific contexts.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Software code generation using Large Language Models (LLMs) is one of the most successful applications of modern artificial intelligence. Foundational models are very effective for popular frameworks that benefit from documentation, examples, and strong community support. In contrast, specialized scientific libraries often lack these resources and may expose unstable APIs under active development, making it difficult for models trained on limited or outdated data. We address these issues for the Gammapy library by developing an agent capable of writing, executing, and validating code in a controlled environment. We present a minimal web demo and an accompanying benchmarking suite. This contribution summarizes the design, reports our current status, and outlines next steps.

Problem

• Developing agent-based code generation for specialized scientific libraries
• Addressing unstable APIs and limited training data in Gammapy
• Creating controlled environment for writing, executing, and validating code

Innovation

• Agent-based code generation for specialized scientific libraries
• Writing, executing, validating code in controlled environment
• Minimal web demo with benchmarking suite for evaluation