System for systematic literature review using multiple AI agents: Concept and an empirical evaluation

2024-03-13

arXiv.org

AI Summary

Existing systematic literature review (SLR) tools support only isolated, single-stage tasks and lack end-to-end automation, heavily relying on manual effort. Method: We propose the first multi-agent AI system for the full SLR lifecycle, built upon large language models (LLMs). It orchestrates specialized agents—leveraging prompt engineering, semantic matching, rule-based filtering, and structured output generation—to automate query formulation, title/abstract-level paper screening, summary synthesis, and research-question-driven deep Q&A analysis. The system enables natural-language interaction and provides interpretable, auditable decision traces. Contribution/Results: Evaluated by ten software engineering experts, our system significantly improves SLR efficiency and inter-reviewer consistency while achieving high user satisfaction. The implementation is open-sourced to facilitate academic replication and extensibility.

Technology Category

• Multiagent Systems: Teamwork
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Systematic Literature Reviews (SLRs) have become the foundation of evidence-based studies, enabling researchers to identify, classify, and combine existing studies based on specific research questions. Conducting an SLR is largely a manual process. Over the previous years, researchers have made significant progress in automating certain phases of the SLR process, aiming to reduce the effort and time needed to carry out high-quality SLRs. However, there is still a lack of AI agent-based models that automate the entire SLR process. To this end, we introduce a novel multi-AI agent model designed to fully automate the process of conducting an SLR. By utilizing the capabilities of Large Language Models (LLMs), our proposed model streamlines the review process, enhancing efficiency and accuracy. The model operates through a user-friendly interface where researchers input their topic, and in response, the model generates a search string used to retrieve relevant academic papers. Subsequently, an inclusive and exclusive filtering process is applied, focusing on titles relevant to the specific research area. The model then autonomously summarizes the abstracts of these papers, retaining only those directly related to the field of study. In the final phase, the model conducts a thorough analysis of the selected papers concerning predefined research questions. We also evaluated the proposed model by sharing it with ten competent software engineering researchers for testing and analysis. The researchers expressed strong satisfaction with the proposed model and provided feedback for further improvement. The code for this project can be found on the GitHub repository at https://github.com/GPT-Laboratory/SLR-automation.

Problem

• Automating the entire systematic literature review workflow using AI agents
• Reducing manual effort and time required for comprehensive literature reviews
• Enhancing efficiency and accuracy of literature reviews through multi-agent AI

Innovation

• Multi-AI-agent system automates entire SLR workflow
• Leverages LLMs to streamline review for efficiency
• User-friendly interface specifies topic for paper analysis