Predicting Empirical AI Research Outcomes with Language Models

2025-06-01

AI Summary

This study addresses the high empirical validation cost of AI research ideas by introducing the first benchmark for predicting the empirical efficacy of novel AI research proposals. Methodologically, it pioneers the use of a fine-tuned GPT-4.1 language model to estimate the success probability of untested ideas, integrating retrieval-augmented paper grounding with expert comparative evaluation. Robustness is assessed across both NLP-specific and cross-domain tasks. Key contributions include: (1) a standardized pairwise relative performance evaluation framework for dual research ideas; (2) empirical validation of generalization to novel, unpublished, and AI-generated ideas; and (3) strong predictive accuracy—64.4% on the NLP subset (exceeding human experts by 15.5 percentage points), 77.0% on the full benchmark, and 63.6% on AI-generated ideas—demonstrating viability as a reward model for idea generation.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Many promising-looking ideas in AI research fail to deliver, but their validation takes substantial human labor and compute. Predicting an idea's chance of success is thus crucial for accelerating empirical AI research, a skill that even expert researchers can only acquire through substantial experience. We build the first benchmark for this task and compare LMs with human experts. Concretely, given two research ideas (e.g., two jailbreaking methods), we aim to predict which will perform better on a set of benchmarks. We scrape ideas and experimental results from conference papers, yielding 1,585 human-verified idea pairs published after our base model's cut-off date for testing, and 6,000 pairs for training. We then develop a system that combines a fine-tuned GPT-4.1 with a paper retrieval agent, and we recruit 25 human experts to compare with. In the NLP domain, our system beats human experts by a large margin (64.4% v.s. 48.9%). On the full test set, our system achieves 77% accuracy, while off-the-shelf frontier LMs like o3 perform no better than random guessing, even with the same retrieval augmentation. We verify that our system does not exploit superficial features like idea complexity through extensive human-written and LM-designed robustness tests. Finally, we evaluate our system on unpublished novel ideas, including ideas generated by an AI ideation agent. Our system achieves 63.6% accuracy, demonstrating its potential as a reward model for improving idea generation models. Altogether, our results outline a promising new direction for LMs to accelerate empirical AI research.

Problem

• Predicting AI research idea success to save resources
• Comparing language models with human expert predictions
• Developing benchmarks for empirical AI research outcomes

Innovation

• Fine-tuned GPT-4.1 with retrieval agent
• Benchmark with human-verified idea pairs
• Robustness tests for non-superficial features