Evaluating LLM-generated code for domain-specific languages: molecular dynamics with LAMMPS

2026-03-20

AI Summary

This work addresses the lack of effective evaluation methods for assessing the scientific validity of domain-specific language (DSL) code—such as LAMMPS molecular dynamics input scripts—generated by large language models (LLMs). To tackle this challenge, the authors propose a lightweight validation framework that combines input file normalization, an extensible DSL parser, and static syntactic and semantic checks. This approach enables domain experts to efficiently verify LLM-generated outputs without requiring deep expertise in the target DSL. By circumventing costly runtime execution, the framework facilitates systematic benchmarking of mainstream LLMs on scientific DSL generation tasks, revealing their current limitations. The study thus provides a practical pathway toward the safe integration of LLMs into specialized scientific computing workflows.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Application Domains: Humanities & Computational Social Science

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large language models (LLMs) are changing the way researchers interact with code and data in scientific computing. While their ability to generate general-purpose code is well established, their effectiveness in producing scientifically valid code/input scripting for domain-specific languages (DSLs) remains largely unexplored. We propose an evaluation procedure that enables domain experts (who may not be experts in the DSL) to assess the validity of LLM-generated input files for LAMMPS, a widely used molecular dynamics (MD) code, and to use those assessments to evaluate the performance of state-of-the-art LLMs and identify common issues. Key to the evaluation procedure are a normalization step to generate canonical files and an extensible parser for syntax analysis. The following steps isolate common errors without incurring costly tests (in time and computational resources). Once a working input file is generated, LLMs can accelerate verification tests. Our findings highlight limitations of LLMs in generating scientific DSLs and a practical path forward for their integration into domain-specific computational ecosystems by domain experts.

Problem

• large language models
• domain-specific languages
• molecular dynamics
• LAMMPS
• scientific code generation

Innovation

• domain-specific languages
• large language models
• LAMMPS
• code validation
• molecular dynamics