Generative AI for Research Data Processing: Lessons Learnt From Three Use Cases

2024-09-16

IEEE International Conference on e-Science

AI Summary

This study addresses the challenge of automating scientific data processing in scenarios characterized by high task complexity and scarce labeled training data—where conventional rule-based systems and traditional machine learning methods fall short. We propose a generative AI applicability assessment framework and a task-oriented prompt engineering methodology tailored for scientific workflows, incorporating domain-specific prompting, structured output constraints, multi-turn verification, and human-in-the-loop feedback. Evaluated on three real-world scientific tasks—plant species identification, health technology assessment information extraction, and Kickstarter project industry classification—using Claude 3 Opus, our approach achieves F1 scores exceeding 0.89 across all tasks. The work bridges a critical gap in flexible, low-label-cost automation for scientific data curation, delivering a reusable methodological framework and practical implementation guidelines. It advances scientific data governance by establishing a novel paradigm grounded in generative AI–enhanced, human-validated automation.

Technology Category

• Natural Language Processing: Generation
• Humans and AI: Crowd Sourcing and Human Computation
• Data Mining & Knowledge Management: Linked Open Data, Knowledge Graphs & KB Completion

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

There has been enormous interest in generative AI since ChatGPT was launched in 2022. However, there are concerns about the accuracy and consistency of the outputs of generative AI. We have carried out an exploratory study on the application of this new technology in research data processing. We identified tasks for which rule-based or traditional machine learning approaches were difficult to apply, and then performed these tasks using generative AI.We demonstrate the feasibility of using the generative AI model Claude 3 Opus in three research projects involving complex data processing tasks:1)Information extraction: We extract plant species names from historical seedlists (catalogues of seeds) published by botanical gardens.2)Natural language understanding: We extract certain data points (name of drug, name of health indication, relative effectiveness, cost-effectiveness, etc.) from documents published by Health Technology Assessment organisations in the EU.3)Text classification: We assign industry codes to projects on the crowdfunding website Kickstarter.We share the lessons we learnt from these use cases: How to determine if generative AI is an appropriate tool for a given data processing task, and if so, how to maximise the accuracy and consistency of the results obtained.

Problem

• Evaluating generative AI's accuracy in research data processing
• Applying generative AI to complex tasks like information extraction
• Maximizing output accuracy in generative AI for specialized tasks

Innovation

• Using Claude 3 Opus for complex data tasks
• Extracting plant species from historical seedlists
• Classifying Kickstarter projects with industry codes