FACTors: A New Dataset for Studying the Fact-checking Ecosystem

2025-05-14

AI Summary

Existing fact-checking research is hindered by the absence of ecosystem-level datasets that span long timeframes, encompass multiple authoritative sources, and adhere to consensus principles. Method: We introduce FACTors—the first ecosystem-level fact-checking dataset—covering 118,112 verified claims issued by 39 IFCN/EFCSN-certified organizations from 1995 to 2025, including 7,327 cross-organizational overlapping claims. We propose the first holistic modeling framework for the fact-checking ecosystem, integrating multi-organizational claim alignment, political bias quantification, and dynamic credibility scoring, powered by web crawling, structured metadata extraction, and statistical evaluation algorithms. Contribution: We release the first open, standardized, and temporally complete ecosystem-level dataset; conduct the first systematic ecological statistical analysis, revealing institutional distribution heterogeneity, sparse inter-organizational collaboration, and latent bias patterns—establishing foundational infrastructure for automated fact-checking and ecosystem governance.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Application Domains: Misinformation & Fake News
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Web Mining and Content Analysis: Web data provenance, reliability, and authenticity
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Data transparency and provenance

Abstract

Our fight against false information is spearheaded by fact-checkers. They investigate the veracity of claims and document their findings as fact-checking reports. With the rapid increase in the amount of false information circulating online, the use of automation in fact-checking processes aims to strengthen this ecosystem by enhancing scalability. Datasets containing fact-checked claims play a key role in developing such automated solutions. However, to the best of our knowledge, there is no fact-checking dataset at the ecosystem level, covering claims from a sufficiently long period of time and sourced from a wide range of actors reflecting the entire ecosystem that admittedly follows widely-accepted codes and principles of fact-checking. We present a new dataset FACTors, the first to fill this gap by presenting ecosystem-level data on fact-checking. It contains 118,112 claims from 117,993 fact-checking reports in English (co-)authored by 1,953 individuals and published during the period of 1995-2025 by 39 fact-checking organisations that are active signatories of the IFCN (International Fact-Checking Network) and/or EFCSN (European Fact-Checking Standards Network). It contains 7,327 overlapping claims investigated by multiple fact-checking organisations, corresponding to 2,977 unique claims. It allows to conduct new ecosystem-level studies of the fact-checkers (organisations and individuals). To demonstrate the usefulness of FACTors, we present three example applications, including a first-of-its-kind statistical analysis of the fact-checking ecosystem, examining the political inclinations of the fact-checking organisations, and attempting to assign a credibility score to each organisation based on the findings of the statistical analysis and political leanings. Our methods for constructing FACTors are generic and can be used to maintain a live dataset that can be updated dynamically.

Problem

• Lack of ecosystem-level fact-checking dataset covering long periods
• Need for diverse sources reflecting entire fact-checking ecosystem
• Absence of datasets supporting automation in fact-checking scalability

Innovation

• Introduces FACTors dataset for ecosystem-level fact-checking studies
• Includes 118,112 claims from 39 IFCN/EFCSN organizations
• Enables dynamic updates and live dataset maintenance