Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews

2026-08-13

AI Summary

This study addresses class imbalance and batch processing effects encountered by large language models (LLMs) in systematic review screening. Through five binary classification experiments, we systematically compared batch versus individual processing modes and evaluated the impact of prevalence metadata. Results indicate that providing prevalence metadata yields limited performance improvements, whereas batch processing significantly alters model decision-making behavior in a manner modulated by class ratios. Crucially, this work reveals discrepancies between aggregate-level and individual-sample analyses, elucidating the underlying mechanisms through which batch processing influences LLM screening decisions. These findings provide essential empirical evidence for optimizing automated screening workflows, highlighting the need to account for processing-induced behavioral shifts when deploying LLMs in evidence synthesis tasks where class distributions are typically skewed.

Abstract

This study analyses LLMs in imbalanced binary classification, using study screening in systematic reviews as the application domain. An experiment was conducted in five reviews, comparing individual and batch processing, with and without prevalence metadata. The results indicate a limited influence of the prevalence metadata, with no evidence that it improves performance. In contrast, batch processing produced larger behavioral changes that varied according to the prevalence of the class. The aggregate and item-level analyses did not always coincide. Therefore, batch processing should be evaluated not only in terms of cost, but also in relation to its effects on decision-making behavior.

Problem

• Class Imbalance
• Batch Effects
• LLM-Based Screening
• Systematic Reviews

Innovation

• LLM-based screening
• Class imbalance
• Batch processing effects
• Decision-making behavior
• Systematic reviews