Understanding LLMs in Title-Abstract Screening: From Disagreements to Recommendations

2026-06-16

AI Summary

Large language models (LLMs) exhibit insufficient reliability in title–abstract screening for systematic literature reviews, yet the reasons for their inconsistency with human experts remain unclear. This study presents the first systematic analysis of discrepancies between LLMs and domain experts across six software engineering reviews, examining over 1,000 papers. Combining Kappa agreement scores (ranging from 0.52 to 0.77) with qualitative methods, we identify recurring failure modes—including ambiguous term boundaries, overreliance on keywords, and misinterpretation of topical relevance. Building on these insights, we propose actionable strategies to enhance LLM trustworthiness in academic screening tasks: pre-deployment semantic validation, parallel multi-model screening, and targeted evaluation focused on boundary cases. Our findings offer both empirical evidence and practical guidance for improving the fidelity and reliability of LLM-assisted systematic reviews.

Abstract

Several studies have examined the use of large language models (LLMs) for title-abstract screening in systematic reviews (SRs), reporting mixed accuracy. However, questions of reliability remain largely unaddressed. In this study, we go beyond quantitative LLM-human agreement metrics and qualitatively investigate how and why LLMs fail. We also propose actionable recommendations. We analyzed disagreements between LLMs and researchers across six software engineering SRs and over 1,000 primary study papers. For each SR, papers were screened independently by human experts and LLMs in zero-shot mode, resulting in Kappa values ranging from 0.52 to 0.77. Qualitative analysis suggests that human-LLM disagreement results from recurring, identifiable causes, such as boundary ambiguity in key terms, keyword overemphasization, and incorrect topic inference. Based on these findings, we propose recommendations such as validating semantic understanding before deployment, running multiple LLMs, and focusing validation efforts on borderline cases. Future studies are needed to validate the impact of our recommendations, and community efforts are needed to develop normative guidelines on LLM usage in SRs.

Problem

• large language models
• systematic reviews
• title-abstract screening
• human-LLM disagreement
• reliability

Innovation

• large language models
• systematic reviews
• qualitative analysis
• human-LLM disagreement
• title-abstract screening