From Verification Burden to Trusted Collaboration: Design Goals for LLM-Assisted Literature Reviews

2025-12-12

AI Summary

Prior LLM applications in literature review suffer from low output credibility, high manual verification costs, and fragmented toolchains. Method: Through interdisciplinary user studies, we distilled core requirements and formulated six actionable design goals. We propose a novel “stepwise-verifiable” human-AI collaboration framework integrating generative explanations, human-AI feedback alignment, and citation-network visualization—transforming LLMs from opaque tools into trustworthy collaborators. Our approach combines qualitative analysis, interactive visual design, and generation-guided explanatory mechanisms to derive design principles and a high-level architecture tailored to real-world scholarly review tasks. Contribution/Results: Evaluation demonstrates that our framework significantly reduces verification effort, enhances output credibility, and improves collaborative efficiency—establishing a foundation for reliable, human-centered LLM-assisted literature synthesis.

Technology Category

• Humans and AI: Crowd Sourcing and Human Computation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large Language Models (LLMs) are increasingly embedded in academic writing practices. Although numerous studies have explored how researchers employ these tools for scientific writing, their concrete implementation, limitations, and design challenges within the literature review process remain underexplored. In this paper, we report a user study with researchers across multiple disciplines to characterize current practices, benefits, and extit{pain points} in using LLMs to investigate related work. We identified three recurring gaps: (i) lack of trust in outputs, (ii) persistent verification burden, and (iii) requiring multiple tools. This motivates our proposal of six design goals and a high-level framework that operationalizes them through improved related papers visualization, verification at every step, and human-feedback alignment with generation-guided explanations. Overall, by grounding our work in the practical, day-to-day needs of researchers, we designed a framework that addresses these limitations and models real-world LLM-assisted writing, advancing trust through verifiable actions and fostering practical collaboration between researchers and AI systems.

Problem

• Addresses trust gaps in LLM outputs for literature reviews
• Reduces verification burden in LLM-assisted academic writing
• Integrates multiple tools into a unified collaborative framework

Innovation

• Enhanced visualization for related papers in literature reviews
• Step-by-step verification integrated into the LLM workflow
• Human feedback aligned with AI-generated explanatory guidance