Can LLMs Automate Fact-Checking Article Writing?

2025-03-22

AI Summary

Current automated fact-checking systems produce outputs lacking readability and communicability, rendering them inadequate substitutes for human-written, comprehensive fact-checking articles. To address this, we introduce the novel task of “public-facing fact-checking article generation” and propose QRAFT, a multi-step reasoning agent framework integrating retrieval augmentation, phased content planning, and fact-aligned generation to emulate professional fact-checkers’ writing processes. Drawing on expert interviews, we establish three core quality criteria: readability, factual accuracy, and structural completeness. Human evaluation demonstrates that QRAFT significantly outperforms mainstream text-generation baselines; although it does not yet match human-expert performance, it establishes the first benchmark dataset and evaluation paradigm for this task—paving a new pathway for leveraging large language models to enhance trustworthy information dissemination.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Automatic fact-checking aims to support professional fact-checkers by offering tools that can help speed up manual fact-checking. Yet, existing frameworks fail to address the key step of producing output suitable for broader dissemination to the general public: while human fact-checkers communicate their findings through fact-checking articles, automated systems typically produce little or no justification for their assessments. Here, we aim to bridge this gap. We argue for the need to extend the typical automatic fact-checking pipeline with automatic generation of full fact-checking articles. We first identify key desiderata for such articles through a series of interviews with experts from leading fact-checking organizations. We then develop QRAFT, an LLM-based agentic framework that mimics the writing workflow of human fact-checkers. Finally, we assess the practical usefulness of QRAFT through human evaluations with professional fact-checkers. Our evaluation shows that while QRAFT outperforms several previously proposed text-generation approaches, it lags considerably behind expert-written articles. We hope that our work will enable further research in this new and important direction.

Problem

• Automate fact-checking article writing using LLMs
• Bridge gap between automated assessments and human-readable outputs
• Develop LLM-based framework mimicking human fact-checkers' workflow

Innovation

• LLM-based agentic framework mimics human workflow
• Extends fact-checking pipeline with article generation
• QRAFT outperforms previous text-generation approaches