SciImpact: A Multi-Dimensional, Multi-Field Benchmark for Scientific Impact Prediction

2026-04-18

AI Summary

Current assessments of scientific impact predominantly rely on citation-based metrics, which inadequately capture the multifaceted and cross-disciplinary nature of real-world influence. To address this limitation, this work proposes SciImpact—the first multidimensional benchmark for scientific impact prediction that integrates heterogeneous data sources, including citations, awards, media coverage, patent citations, and policy or practice adoption, across both short- and long-term horizons. Spanning 19 scientific domains and comprising 215,928 paper pairs with significant impact disparities, SciImpact enables systematic evaluation of large language models through heterogeneous data fusion, targeted web crawling, contrastive sample construction, and multi-task fine-tuning. Experiments demonstrate that a fine-tuned open-source 4B-parameter model substantially outperforms larger closed-source counterparts, including 30B-parameter models and o1-mini, thereby validating the benchmark’s challenge and efficacy.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Application Domains: Humanities & Computational Social Science
• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The rapid growth of scientific literature calls for automated methods to assess and predict research impact. Prior work has largely focused on citation-based metrics, leaving limited evaluation of models' capability to reason about other impact dimensions. To this end, we introduce SciImpact, a large-scale, multi-dimensional benchmark for scientific impact prediction spanning 19 fields. SciImpact captures various forms of scientific influence, ranging from citation counts to award recognition, media attention, patent reference, and artifact adoption, by integrating heterogeneous data sources and targeted web crawling. It comprises 215,928 contrastive paper pairs reflecting meaningful impact differences in both short-term (e.g., Best Paper Award) and long-term settings (e.g., Nobel Prize). We evaluate 11 widely used large language models (LLMs) on SciImpact. Results show that off-the-shelf models exhibit substantial variability across dimensions and fields, while multi-task supervised fine-tuning consistently enables smaller LLMs (e.g., 4B) to markedly outperform much larger models (e.g., 30B) and surpass powerful closed-source LLMs (e.g., o4-mini). These results establish SciImpact as a challenging benchmark and demonstrate its value for multi-dimensional, multi-field scientific impact prediction. Our project homepage is https://flypig23.github.io/sciimpact-homepage/

Problem

• scientific impact prediction
• multi-dimensional evaluation
• citation-based metrics
• heterogeneous impact indicators
• large language models

Innovation

• scientific impact prediction
• multi-dimensional benchmark
• large language models
• contrastive paper pairs
• multi-task fine-tuning