LLM-Assisted Empirical Software Engineering: Systematic Literature Review and Research Agenda

2026-04-28

AI Summary

Empirical software engineering faces significant challenges, including large-scale data, methodological complexity, and poor reproducibility, while the application of large language models (LLMs) in this domain lacks systematic integration. This study conducts a systematic literature review of 50 studies published between 2020 and 2025 across 12 leading conferences and journals, offering the first comprehensive taxonomy of 69 LLM-supported auxiliary tasks in empirical software engineering. The analysis reveals that LLMs are predominantly employed in data processing and analysis phases to enhance automation, efficiency, and scalability. However, their use is frequently hindered by issues such as hallucination, inconsistent outputs, sensitivity to prompting, insufficient reporting of reproducibility, and a notable absence of human-centered collaboration and transparency. Building on these findings, this work proposes a research agenda oriented toward the responsible application of LLMs in empirical software engineering.

Abstract

Context: Empirical Software Engineering (ESE) faces increasing challenges due to data scale, methodological complexity, and reproducibility concerns. Large Language Models (LLMs) have emerged as promising tools to support empirical workflows, yet their use remains fragmented, with no comprehensive synthesis to guide responsible adoption. Aims: This study analyzes how LLMs are used in ESE, examining supported tasks, phases of the empirical lifecycle, integration into workflows, reported benefits and limitations, and the extent of reproducibility-related reporting. It also identifies gaps and future research directions. Method: We conducted a systematic literature review of peer-reviewed papers (2020-2025) across 12 leading software engineering venues, resulting in 50 primary studies analyzed through qualitative and quantitative synthesis. Results: We identified 69 LLM-assisted tasks, mainly in mining software repositories and controlled experiments, focusing on classification, filtering, and evaluation. LLMs are used across multiple phases but are concentrated in data processing and analysis. Their integration is largely automation-oriented, with limited decision-support use. Benefits emphasize efficiency and scalability, while limitations include hallucinations, inconsistency, prompt sensitivity, and reproducibility issues. Reporting practices are often incomplete. Conclusion: LLM use in ESE is growing but remains automation-driven, with gaps in human-centered integration and transparency. We outline implications and research agenda for responsible use.

Problem

• Empirical Software Engineering
• Large Language Models
• Reproducibility
• Methodological Complexity
• Data Scale

Innovation

• Large Language Models
• Empirical Software Engineering
• Systematic Literature Review
• Reproducibility
• Human-AI Collaboration