Literature Meets Data: A Synergistic Approach to Hypothesis Generation

2024-10-22

arXiv.org

AI Summary

This study addresses the narrow scope of purely theory- or data-driven approaches in AI-assisted innovation by proposing the first LLM-based hypothesis generation framework that jointly leverages scholarly literature and empirical data. Methodologically, it introduces a novel dual-source synergy mechanism integrating literature semantic parsing with multi-source data alignment, augmented by domain-specific prompt engineering and rigorous human evaluation. Key contributions include: (1) establishing the first theory- and data-coordinated paradigm for automated hypothesis generation; (2) achieving statistically significant improvements in hypothesis quality—+8.97% over few-shot baselines, +15.75% over literature-only methods, and +3.37% over data-only methods—across five benchmark datasets; and (3) demonstrating via human evaluation that the framework substantially enhances decision-making accuracy in AI-content identification tasks, with gains ranging from 7.44% to 14.19%.

Technology Category

• Humans and AI: Understanding People, Theories, Concepts and Methods
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

AI holds promise for transforming scientific processes, including hypothesis generation. Prior work on hypothesis generation can be broadly categorized into theory-driven and data-driven approaches. While both have proven effective in generating novel and plausible hypotheses, it remains an open question whether they can complement each other. To address this, we develop the first method that combines literature-based insights with data to perform LLM-powered hypothesis generation. We apply our method on five different datasets and demonstrate that integrating literature and data outperforms other baselines (8.97% over few-shot, 15.75% over literature-based alone, and 3.37% over data-driven alone). Additionally, we conduct the first human evaluation to assess the utility of LLM-generated hypotheses in assisting human decision-making on two challenging tasks: deception detection and AI generated content detection. Our results show that human accuracy improves significantly by 7.44% and 14.19% on these tasks, respectively. These findings suggest that integrating literature-based and data-driven approaches provides a comprehensive and nuanced framework for hypothesis generation and could open new avenues for scientific inquiry.

Problem

• Artificial Intelligence
• Knowledge Integration
• Decision Making

Innovation

• Integrative Knowledge Approach
• Large Language Models
• Enhanced Human Decision-making