Toward Reliable Ad-hoc Scientific Information Extraction: A Case Study on Two Materials Datasets

2024-06-08

Annual Meeting of the Association for Computational Linguistics

AI Summary

This study evaluates GPT-4’s capability to extract structured information on-demand from materials science literature, specifically assessing its zero-shot fidelity in reproducing two manually curated materials datasets. Methodologically, we employ an interdisciplinary, domain-expert–driven error attribution framework—integrated with rigorous human annotation—to systematically diagnose model output deviations across dimensions including numerical accuracy, contextual disambiguation, and unit standardization. Results reveal significant fidelity bottlenecks in GPT-4’s scientific information extraction (IE), particularly in precision-critical tasks. Our key contribution is the first application of deep expert-led error analysis to scientific IE evaluation, enabling quantitative characterization of large language models’ reliability boundaries in authentic research settings. We further propose a scalable, expert-validated benchmark for scientific IE assessment, advancing methodological foundations for high-fidelity AI-assisted scientific discovery.

Technology Category

• Natural Language Processing: Information Extraction
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity

Abstract

We explore the ability of GPT-4 to perform ad-hoc schema based information extraction from scientific literature. We assess specifically whether it can, with a basic prompting approach, replicate two existing material science datasets, given the manuscripts from which they were originally manually extracted. We employ materials scientists to perform a detailed manual error analysis to assess where the model struggles to faithfully extract the desired information, and draw on their insights to suggest research directions to address this broadly important task.

Problem

• Assessing GPT-4's ability for schema-based scientific information extraction
• Evaluating replication of material science datasets via basic prompting
• Identifying model limitations through manual error analysis by experts

Innovation

• GPT-4 for ad-hoc schema extraction
• Basic prompting replicates material datasets
• Manual error analysis guides improvements