Large Language Models as Supervised Extraction Assistants: Lowering the Barrier to Documentation Standard Adoption in Agent-Based Modelling

2026-06-11

AI Summary

This study addresses the widespread lack of adherence to reporting standards such as RAT-RS in agent-based modelling (ABM) research, largely due to the time-consuming and undervalued nature of manual reporting. It presents the first systematic evaluation of the feasibility of using large language models (LLMs) to automate the generation of RAT-RS–compliant content. The authors propose a practical framework based on supervised information extraction and introduce heuristic rules to delineate model reliability and the boundaries requiring human intervention. Experimental comparisons across four LLMs demonstrate that these models produce coherent and reliable outputs for descriptive tasks, significantly enhancing reporting quality and consistency. However, they exhibit notable limitations in explanatory and evaluative tasks, underscoring the continued need for human oversight in more interpretive aspects of ABM reporting.

Abstract

Agent-Based Modelling (ABM) relies on clear documentation to ensure credibility and transparency. Although standards exist for documenting models (e.g. ODD), processes (e.g. TRACE, EABSS), and data use (e.g. RAT-RS), their adoption remains limited due to the effort required to produce documentation that is often treated as supplementary. This paper explores the use of Large Language Models (LLMs) to facilitate and partially automate such processes. We conduct a feasibility study focusing on the underused Rigour and Transparency Reporting Standard (RAT-RS), using four LLMs to extract reports from a published ABM paper. We assess consistency and performance across question types, finding that LLMs generate coherent outputs and perform more reliably on descriptive than on explanatory or evaluative tasks. While LLMs can improve reporting quality and consistency, they also exhibit notable limitations. We identify practical heuristics for when LLM-assisted documentation is reliable and when human oversight is needed and call for systematic community-level exploration to enhance rigour and adoption in ABM reporting.

Problem

• Agent-Based Modelling
• Documentation Standards
• RAT-RS
• Adoption Barrier
• Scientific Reporting

Innovation

• Large Language Models
• Agent-Based Modelling
• Documentation Standards
• RAT-RS
• Automated Reporting