CRISP: Characterizing Relative Impact of Scholarly Publications

2026-03-25

AI Summary

This work addresses the limitation of existing approaches that evaluate cited references in isolation, making it difficult to assess their relative influence within the same citing document. To overcome this, the authors propose CRISP, a novel method that introduces, for the first time, a large language model–based joint ranking mechanism. By randomly permuting the order of all cited references three times and aggregating results via majority voting, CRISP effectively mitigates positional bias inherent in language models. Integrated with citation context analysis, the approach achieves a 9.5% improvement in accuracy and an 8.3% gain in F1 score on a human-annotated dataset, while simultaneously reducing the number of large model invocations, thereby offering both computational efficiency and scalability.

Abstract

Assessing a cited paper's impact is typically done by analyzing its citation context in isolation within the citing paper. While this focuses on the most directly relevant text, it prevents relative comparisons across all the works a paper cites. We propose CRISP, which instead jointly ranks all cited papers within a citing paper using large language models (LLMs). To mitigate LLMs' positional bias, we rank each list three times in a randomized order and aggregate the impact labels through majority voting. This joint approach leverages the full citation context, rather than evaluating citations independently, to more reliably distinguish impactful references. CRISP outperforms a prior state-of-the-art impact classifier by +9.5% accuracy and +8.3% F1 on a dataset of human-annotated citations. CRISP further gains efficiency through fewer LLM calls and performs competitively with an open-source model, enabling scalable, cost-effective citation impact analysis. We release our rankings, impact labels, and codebase to support future research.

Problem

• citation impact
• relative comparison
• scholarly publications
• citation context
• impact assessment

Innovation

• citation impact
• large language models
• joint ranking
• positional bias mitigation
• relative comparison