Measuring the Gap Between Human and LLM Research Ideas

2026-07-01

AI Summary

This study addresses the lack of systematic, quantitative analysis comparing scientific idea generation by large language models (LLMs) and human researchers. The authors construct the first large-scale evaluation framework that extracts inspirational antecedents from high-quality human-authored papers via reverse engineering to prompt LLMs to generate novel research ideas. They introduce a dual-axis taxonomy—spanning “opportunity modes” and “research paradigms”—to characterize and compare these ideas. Experimental results reveal that LLM-generated ideas are significantly concentrated in bridging-type opportunities and integrative methodologies, whereas human-generated ideas exhibit broader coverage and greater diversity. This disparity highlights inherent limitations and systematic biases in the creative scope of current LLMs within scientific ideation contexts.

Abstract

LLMs are increasingly used to brainstorm research ideas, but existing evaluations mostly judge individual ideas by novelty, feasibility, or expert preference. We instead ask: how far are current LLM-generated ideas from human researchers? To characterize this gap, we build a large-scale evaluation framework for ideation from high-quality human research papers. For each paper, we reverse-engineer a small set of closely related prior works that likely inspired its core idea. LLMs are then prompted to generate a new idea from the set of paper titles and summaries. We introduce a two-axis research-taste taxonomy to profile each idea by its opportunity pattern and research paradigm, and use it to quantify the divergence between human and LLM ideas. Across idea sets generated by different LLMs, we observe a consistent distributional gap: LLM ideas are disproportionately concentrated around bridge-like opportunities and synthesis methods, whereas the human paper reference distribution spreads more broadly across ways of framing gaps and constructing contributions. This result suggests that strong LLMs can produce a range of reasonable ideas, but that range remains narrower than, and systematically shifted relative to, human research taste.

Problem

• LLM-generated ideas
• human research ideas
• research taste
• ideation gap
• idea evaluation

Innovation

• research ideation
• LLM evaluation
• research-taste taxonomy
• opportunity pattern
• distributional gap