Know Your Source: A Public Knowledge Store for Media Background Checks

2026-07-02

AI Summary

This work addresses a critical limitation in current retrieval-augmented generation (RAG) fact-checking systems, which typically assume retrieved evidence is reliable while overlooking that sources may be biased, unreliable, or outdated. To mitigate this, the authors introduce MEDIAREF—the first publicly available and updatable media background knowledge base, encompassing 200 news outlets. Constructed through web crawling and document aggregation, MEDIAREF enables Media Background Checking (MBC) via large language models without reliance on costly proprietary APIs. Experimental results demonstrate that integrating MEDIAREF significantly enhances models’ ability to assess source credibility, thereby improving the transparency and reliability of fact-checking systems. Furthermore, MEDIAREF advances reproducible research and equitable evaluation in source-critical reasoning.

Abstract

LLM-based retrieval-augmented generation (RAG) is increasingly used for automated fact-checking (AFC) and related tasks. By grounding LLM outputs in retrieved evidence, RAG-based systems provide transparent justifications while allowing external information to be updated independently of the underlying model. However, existing approaches often assume retrieved evidence is reliable, although real-world information may be conflicting, outdated, and can originate from unreliable or biased sources. Recent work on *source-critical reasoning* addresses this challenge through media background checks (MBCs) (Schlichtkrull, 2024), which assess the credibility of evidence sources to support downstream fact verification. However, generating MBCs relies on costly proprietary search APIs, limiting reproducibility. To mitigate this issue, we introduce MEDIAREF, a publicly available knowledge store of web-sourced documents that enables reproducible, low-cost evaluation of MBC generation across 200 media sources. We describe a reproducible methodology for constructing and updating the collection, assess widely used LLMs on the MBC generation task, and demonstrate that MEDIAREF supports higher-quality MBC generation through both automatic and qualitative evaluation.

Problem

• retrieval-augmented generation
• automated fact-checking
• source credibility
• media background checks
• reproducibility

Innovation

• Media Background Checks
• Retrieval-Augmented Generation
• Source-Critical Reasoning
• Public Knowledge Store
• Fact-Checking