Content-aware rankings: a new approach to rankings in scholarship

2025-04-07

AI Summary

Current academic institutional rankings overemphasize publication counts while neglecting the semantic meaning of citations. To address this, we propose a citation-intent-driven ranking paradigm—the first to incorporate SciTe’s three-category citation semantics (supporting, contrasting, and mentioning) into scholarly evaluation—thereby establishing a content-oriented, interpretable, and fine-grained quality-weighted ranking framework. Methodologically, we integrate NLP and semantic analysis techniques to design a citation-intent-aware weighted aggregation algorithm grounded in structured citation classification data. Empirical evaluation across multiple disciplines, journals, and universities demonstrates that our approach effectively discriminates between distinct impact types—such as theoretical leadership versus technical application—and exhibits superior discriminative power and robustness. This work advances scholarly assessment from a purely quantity-based model toward a dual-dimensional “quality–intent” paradigm.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Entity rankings (e.g., institutions, journals) are a core component of academia and related industries. Existing approaches to institutional rankings have relied on a variety of data sources, and approaches to computing outcomes, but remain controversial. One limitation of existing approaches is reliance on scholarly output (e.g., number of publications associated with a given institution during a time period). We propose a new approach to rankings - one that relies not on scholarly output, but rather on the type of citations received (an implementation of the Scite Index). We describe how the necessary data can be gathered, as well as how relevant metrics are computed. To demonstrate the utility of our approach, we present rankings of fields, journals, and institutions, and discuss the various ways Scite's data can be deployed in the context of rankings. Implications, limitations, and future directions are discussed.

Problem

• Existing institutional rankings rely on controversial scholarly output metrics
• Proposing a new ranking method based on citation types (Scite Index)
• Demonstrating utility via field, journal, and institution rankings

Innovation

• Uses citation types instead of output quantity
• Implements Scite Index for ranking metrics
• Generates rankings for fields, journals, institutions