On the Use of a Large Language Model to Support the Conduction of a Systematic Mapping Study: A Brief Report from a Practitioner's View

2026-02-09

AI Summary

This study addresses the time-consuming and subjectivity-prone nature of manual text data processing in systematic mapping studies, as well as the lack of practical guidance for end-to-end integration of large language models (LLMs). The authors propose an LLM-assisted, end-to-end approach that combines structured prompt engineering with human verification, spanning critical stages such as literature screening and data extraction. Findings indicate that this method substantially improves processing efficiency and data standardization, while also highlighting challenges including high prompt development costs and model hallucinations. The work contributes empirical insights into the use of LLMs for evidence synthesis, elucidates the trade-offs between efficiency gains and methodological risks, and offers actionable, reusable recommendations for practitioners.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The use of Large Language Models (LLMs) has drawn growing interest within the scientific community. LLMs can handle large volumes of textual data and support methods for evidence synthesis. Although recent studies highlight the potential of LLMs to accelerate screening and data extraction steps in systematic reviews, detailed reports of their practical application throughout the entire process remain scarce. This paper presents an experience report on the conduction of a systematic mapping study with the support of LLMs, describing the steps followed, the necessary adjustments, and the main challenges faced. Positive aspects are discussed, such as (i) the significant reduction of time in repetitive tasks and (ii) greater standardization in data extraction, as well as negative aspects, including (i) considerable effort to build reliable well-structured prompts, especially for less experienced users, since achieving effective prompts may require several iterations and testing, which can partially offset the expected time savings, (ii) the occurrence of hallucinations, and (iii) the need for constant manual verification. As a contribution, this work offers lessons learned and practical recommendations for researchers interested in adopting LLMs in systematic mappings and reviews, highlighting both efficiency gains and methodological risks and limitations to be considered.

Problem

• Large Language Models
• Systematic Mapping Study
• Evidence Synthesis
• Prompt Engineering
• Hallucination

Innovation

• Large Language Models
• Systematic Mapping Study
• Evidence Synthesis
• Prompt Engineering
• Research Automation