ProfOlaf: Semi-Automated Tool for Systematic Literature Reviews

2025-10-30

AI Summary

Systematic literature reviews (SLRs) are time-consuming, error-prone, and poorly supported by existing tools, which only automate isolated steps rather than enabling end-to-end intelligent collaboration. This paper proposes a semi-automated SLR framework integrating iterative snowballing retrieval, human-in-the-loop screening, and large language model (LLM)-driven content understanding—including topic modeling, key information extraction, and intelligent question answering. Its core contribution lies in deeply embedding LLMs across the entire SLR workflow to enable semantic-aware literature analysis and dynamic, feedback-driven iteration; reliability is ensured via configurable human verification checkpoints. Empirical evaluation demonstrates that the approach reduces screening and analysis time by 2.3× on average, improves annotation consistency (Cohen’s Kappa increases by 0.32), and enhances procedural reproducibility. The framework has been validated across diverse domains—including software engineering, medicine, and education—confirming its cross-domain efficacy.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Humans and AI: Human-in-the-loop Machine Learning
• Natural Language Processing: Summarization

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Systematic reviews and mapping studies are critical for synthesizing research, identifying gaps, and guiding future work, but they are often labor-intensive and time-consuming. Existing tools provide partial support for specific steps, leaving much of the process manual and error-prone. We present ProfOlaf, a semi-automated tool designed to streamline systematic reviews while maintaining methodological rigor. ProfOlaf supports iterative snowballing for article collection with human-in-the-loop filtering and uses large language models to assist in analyzing articles, extracting key topics, and answering queries about the content of papers. By combining automation with guided manual effort, ProfOlaf enhances the efficiency, quality, and reproducibility of systematic reviews across research fields. A video describing and demonstrating ProfOlaf is available at: https://youtu.be/4noUXfcmxsE

Problem

• Automating labor-intensive systematic literature review processes
• Reducing manual errors in research synthesis and gap identification
• Enhancing efficiency and reproducibility across research fields

Innovation

• Semi-automated tool streamlines systematic literature reviews
• Uses iterative snowballing with human-in-the-loop filtering
• Leverages large language models for article analysis