The Shiny Scary Future of Automated Research Synthesis in HCI

2025-01-27

AI Summary

This paper addresses reliability concerns and human-centered boundaries in the automated application of large language models (LLMs) for systematic literature reviews (SLRs) in Human-Computer Interaction (HCI). Method: Through empirical LLM experiments, human-AI collaborative workflow design, and HCI methodology analysis, it systematically delineates review stages amenable to automation (e.g., initial screening) versus those requiring human agency (e.g., thematic modeling, cross-study inference). Contribution/Results: The study proposes the “human-centered augmentation” ethical framework, rigorously defining LLMs’ capabilities and limitations in research synthesis. It delivers actionable, rigor-preserving guidelines for SLR practitioners and has been selected for a spotlight discussion at CHI ’25.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Intelligent Robots: Human-Robot Interaction

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Automation and semi-automation through computational tools like LLMs are also making their way to deployment in research synthesis and secondary research, such as systematic reviews. In some steps of research synthesis, this has the opportunity to provide substantial benefits by saving time that previously was spent on repetitive tasks. The screening stages in particular may benefit from carefully vetted computational support. However, this position paper argues for additional caution when bringing in such tools to the analysis and synthesis phases, where human judgement and expertise should be paramount throughout the process.

Problem

• LLMs Assistance
• Research Integrity
• Human Expertise Preservation

Innovation

• Large Language Models
• Systematic Review
• Human Expertise Integration