RIGOURATE: Quantifying Scientific Exaggeration with Evidence-Aligned Claim Evaluation

2026-01-07

arXiv.org

AI Summary

Scientific papers often suffer from exaggerated claims that undermine research rigor. This work proposes a two-stage multimodal framework to address this issue: first, a fine-tuned re-ranking model retrieves supporting evidence from the main text, and second, a fine-tuned language model predicts a claim exaggeration score accompanied by an explanation. We construct the first large-scale annotated dataset of claim–evidence pairs, integrating collaborative annotations from multiple large language models, human feedback, and calibration against peer-review comments. Experiments on ICLR and NeurIPS papers demonstrate that our approach significantly outperforms strong baselines in both evidence retrieval and exaggeration detection, enabling—for the first time—a quantifiable assessment of scientific claim rigor and advancing transparency and credibility in scholarly communication.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Scientific rigour tends to be sidelined in favour of bold statements, leading authors to overstate claims beyond what their results support. We present RIGOURATE, a two-stage multimodal framework that retrieves supporting evidence from a paper's body and assigns each claim an overstatement score. The framework consists of a dataset of over 10K claim-evidence sets from ICLR and NeurIPS papers, annotated using eight LLMs, with overstatement scores calibrated using peer-review comments and validated through human evaluation. It employes a fine-tuned reranker for evidence retrieval and a fine-tuned model to predict overstatement scores with justification. Compared to strong baselines, RIGOURATE enables improved evidence retrieval and overstatement detection. Overall, our work operationalises evidential proportionality and supports clearer, more transparent scientific communication.

Problem

• scientific exaggeration
• overstatement
• evidential proportionality
• scientific communication
• claim evaluation

Innovation

• scientific exaggeration
• evidence-aligned claim evaluation
• overstatement detection
• multimodal framework
• evidential proportionality