Multilingual, Multimodal Pipeline for Creating Authentic and Structured Fact-Checked Claim Dataset

2026-01-12

AI Summary

This study addresses critical limitations in existing fact-checking datasets—namely, insufficient multilingual coverage, inadequate multimodal evidence integration, lack of structured annotations, and coarse-grained claim-evidence alignment—which hinder research on interpretable and cross-lingual misinformation detection. To overcome these challenges, the authors propose an end-to-end pipeline that aggregates ClaimReview sources and retrieves full debunking articles, standardizes heterogeneous verdicts, and fuses structured metadata with aligned visual content to construct the first French and German multimodal fact-checking datasets. The work introduces a novel fine-grained evidence categorization scheme coupled with a verdict linkage mechanism, leveraging large language models and multimodal large models to automatically extract evidence and generate explanatory rationales. Evaluations via G-Eval and human assessment confirm that the dataset effectively supports the development of interpretable, evidence-based fact-checking models, establishing a foundation for multilingual, multimodal misinformation research.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data
• Machine Learning: Multimodal Learning

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

The rapid proliferation of misinformation across online platforms underscores the urgent need for robust, up-to-date, explainable, and multilingual fact-checking resources. However, existing datasets are limited in scope, often lacking multimodal evidence, structured annotations, and detailed links between claims, evidence, and verdicts. This paper introduces a comprehensive data collection and processing pipeline that constructs multimodal fact-checking datasets in French and German languages by aggregating ClaimReview feeds, scraping full debunking articles, normalizing heterogeneous claim verdicts, and enriching them with structured metadata and aligned visual content. We used state-of-the-art large language models (LLMs) and multimodal LLMs for (i) evidence extraction under predefined evidence categories and (ii) justification generation that links evidence to verdicts. Evaluation with G-Eval and human assessment demonstrates that our pipeline enables fine-grained comparison of fact-checking practices across different organizations or media markets, facilitates the development of more interpretable and evidence-grounded fact-checking models, and lays the groundwork for future research on multilingual, multimodal misinformation verification.

Problem

• fact-checking
• multilingual
• multimodal
• misinformation
• structured dataset

Innovation

• multilingual fact-checking
• multimodal pipeline
• structured claim dataset
• evidence extraction
• justification generation