Using a Human-AI Teaming Approach to Create and Curate Scientific Datasets with the SCILIRE System

2026-03-13

AI Summary

The exponential growth of scientific literature has rendered manual extraction of structured knowledge increasingly challenging. To address this, this work proposes SCILIRE, a system that innovatively integrates human–machine collaboration deeply into an iterative knowledge extraction pipeline: researchers review and correct outputs generated by large language models, and their interactive feedback is immediately leveraged to refine subsequent inference. By synergistically combining large language models, an interactive feedback mechanism, and an iterative validation workflow, the approach significantly enhances the fidelity of information extraction across multiple scientific domains, thereby enabling efficient construction of high-quality scientific datasets.

Technology Category

• Natural Language Processing: Information Extraction
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The rapid growth of scientific literature has made manual extraction of structured knowledge increasingly impractical. To address this challenge, we introduce SCILIRE, a system for creating datasets from scientific literature. SCILIRE has been designed around Human-AI teaming principles centred on workflows for verifying and curating data. It facilitates an iterative workflow in which researchers can review and correct AI outputs. Furthermore, this interaction is used as a feedback signal to improve future LLM-based inference. We evaluate our design using a combination of intrinsic benchmarking outcomes together with real-world case studies across multiple domains. The results demonstrate that SCILIRE improves extraction fidelity and facilitates efficient dataset creation.

Problem

• scientific literature
• structured knowledge extraction
• dataset creation
• human-AI teaming
• LLM-based inference

Innovation

• Human-AI teaming
• scientific dataset curation
• LLM-based inference
• iterative feedback
• knowledge extraction