RenoBench: A Citation Parsing Benchmark

2026-03-26

AI Summary

This work addresses the limitations of existing citation parsing evaluation methods, which often suffer from poor generalizability, reliance on synthetic data, or lack of public availability, thereby hindering reproducible research. To overcome these challenges, we introduce RenoBench, an open-source benchmark comprising 10,000 high-quality, real-world citations carefully sampled via automated validation and feature-driven selection from a corpus of 161,000 references spanning four major publishing ecosystems. RenoBench is the first large-scale, publicly available citation parsing benchmark that is cross-lingual, cross-platform, and derived entirely from authentic sources, enabling standardized and reproducible evaluation at the field level. Experimental results demonstrate that fine-tuned language models achieve strong performance on this task, significantly advancing both citation parsing and metascientific research.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Accurate parsing of citations is necessary for machine-readable scholarly infrastructure. But, despite sustained interest in this problem, existing evaluation techniques are often not generalizable, based on synthetic data, or not publicly available. We introduce RenoBench, a public domain benchmark for citation parsing, sourced from PDFs released on four publishing ecosystems: SciELO, Redalyc, the Public Knowledge Project, and Open Research Europe. Starting from 161,000 annotated citations, we apply automated validation and feature-based sampling to produce a dataset of 10,000 citations spanning multiple languages, publication types, and platforms. We then evaluate a variety of citation parsing systems and report field-level precision and recall. Our results show strong performance from language models, particularly when fine-tuned. RenoBench enables reproducible, standardized evaluation of citation parsing systems, and provides a foundation for advancing automated citation parsing and metascientific research.

Problem

• citation parsing
• evaluation benchmark
• scholarly infrastructure
• reproducibility
• standardized evaluation

Innovation

• citation parsing
• benchmark dataset
• multilingual scholarly data
• language models
• reproducible evaluation