Highlighting Case Studies in LLM Literature Review of Interdisciplinary System Science

2025-03-16

Applied Informatics

AI Summary

This study evaluates large language models’ (LLMs) capabilities in evidence extraction, citation reproduction, and research question answering within interdisciplinary systems science systematic literature reviews (SLRs). To address the challenge of rigorously assessing LLM-generated scholarly outputs, we propose a hybrid evaluation framework integrating semantic text highlighting visualization with expert collaborative review. We further introduce a novel automated semantic consistency metric based on cosine similarity of Transformer-derived sentence embeddings—demonstrating moderate-to-strong correlation with expert judgments (r = 0.48–0.77). Experimental results show >95% accuracy in citation reproduction and 83% accuracy in answering domain-specific research questions. Our core contribution is a scalable, verifiable dual-track evaluation paradigm that balances human-in-the-loop reliability with automated reproducibility. Crucially, we establish embedding-based semantic similarity as a valid proxy for semantic consistency, providing a methodological foundation for LLM-augmented academic literature analysis.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs) were used to assist four Commonwealth Scientific and Industrial Research Organisation (CSIRO) researchers to perform systematic literature reviews (SLR). We evaluate the performance of LLMs for SLR tasks in these case studies. In each, we explore the impact of changing parameters on the accuracy of LLM responses. The LLM was tasked with extracting evidence from chosen academic papers to answer specific research questions. We evaluate the models' performance in faithfully reproducing quotes from the literature and subject experts were asked to assess the model performance in answering the research questions. We developed a semantic text highlighting tool to facilitate expert review of LLM responses. We found that state of the art LLMs were able to reproduce quotes from texts with greater than 95% accuracy and answer research questions with an accuracy of approximately 83%. We use two methods to determine the correctness of LLM responses; expert review and the cosine similarity of transformer embeddings of LLM and expert answers. The correlation between these methods ranged from 0.48 to 0.77, providing evidence that the latter is a valid metric for measuring semantic similarity.

Problem

• Evaluating LLM performance in systematic literature reviews
• Assessing impact of parameters on LLM response accuracy
• Developing tools for expert review of LLM outputs

Innovation

• LLMs assist systematic literature reviews efficiently
• Semantic text highlighting tool enhances expert review
• Transformer embeddings measure answer similarity effectively