RWGBench: Evaluating Scholarly Positioning in Related Work Generation

2026-05-30

AI Summary

This work addresses the limitations of existing evaluation methods for related work generation, which predominantly rely on text similarity metrics and fail to capture critical issues in scholarly positioning, such as inappropriate citations or misaligned references. The paper reframes related work generation as an academic positioning task and introduces RWGBench, a citation-centered, multi-dimensional evaluation framework encompassing four key dimensions: citation selection, contextual appropriateness, organizational structure, and discursive logic. Constructed from 40,108 computer science papers and over 1 million supporting documents, the benchmark combines automated metrics with expert assessments to validate its efficacy. Experiments demonstrate that RWGBench reveals systematic citation-level shortcomings in current systems and that its novel metrics align more closely with expert judgments than conventional text similarity measures, thereby offering a more academically grounded evaluation standard for related work generation.

Abstract

Large language models have shown strong fluency in scientific writing, yet the evaluation of related work generation (RWG) remains limited. Existing RWG evaluations largely inherit summarization-oriented metrics, using lexical or semantic similarity to reference sections as proxies for quality. However, related work writing is fundamentally a citation-level scholarly positioning task: it requires selecting, organizing, and framing prior work to clarify how a target paper relates to, differs from, and contributes beyond existing research.As a result, models may generate coherent and semantically-relevant text while exhibiting academically critical failures, such as inappropriate citation selection or misplaced references, that conventional metrics do not capture.To this end, we introduce \textbf{RWGBench}, a benchmark that evaluates RWG from the perspective of citation decision-making rather than text similarity. RWGBench is constructed from a large-scale collection of 40,108 computer science papers and a retrieval corpus of 1.09 million documents, with a carefully curated test set comprising 100 papers and their corresponding published related work sections.We propose a multi-dimensional evaluation framework that assesses citation selection, contextual appropriateness, organization, and discourse structure.Experiments reveal systematic limitations in current systems that are obscured by standard evaluations, while Oracle studies further disentangle retrieval-level and generation-level bottlenecks. Human evaluation further shows that our citation-centric metrics align substantially better with expert judgment than surface-level text metrics. RWGBench offers a citation-centric testbed for developing and evaluating related work generation systems that are better aligned with scholarly writing practices.

Problem

• Related Work Generation
• Scholarly Positioning
• Citation Evaluation
• Scientific Writing
• LLM Evaluation

Innovation

• related work generation
• citation-centric evaluation
• scholarly positioning
• RWGBench
• academic writing