Can LLMs Improve Multimodal Fact-Checking by Asking Relevant Questions?

2024-10-06

AI Summary

Current large language models (LLMs) face limitations in multimodal fact-checking due to the scarcity of high-quality, verifiable fact-checking questions (FCQs). This work presents the first systematic validation of LLMs’ capability to autonomously generate relevant and verifiable FCQs. We propose LRQ-FACT, a novel framework that deeply integrates FCQ generation into the multimodal fact-checking pipeline: FCQs are prompted via instruction engineering; textual and visual evidence is jointly retrieved; and question–evidence co-reasoning enables fine-grained factual assessment. Evaluated across multiple mainstream multimodal fact-checking benchmarks, LRQ-FACT consistently outperforms strong baselines, achieving average accuracy gains of 3.2–5.7 percentage points while improving model robustness. Our key contribution is establishing FCQ generation as a critical intermediate task—enabling interpretable, traceable, and fully automated multimodal fact-checking.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Traditional fact-checking relies on humans to formulate relevant and targeted fact-checking questions (FCQs), search for evidence, and verify the factuality of claims. While Large Language Models (LLMs) have been commonly used to automate evidence retrieval and factuality verification at scale, their effectiveness for fact-checking is hindered by the absence of FCQ formulation. To bridge this gap, we seek to answer two research questions: (1) Can LLMs generate relevant FCQs? (2) Can LLM-generated FCQs improve multimodal fact-checking? We therefore introduce a framework LRQ-FACT for using LLMs to generate relevant FCQs to facilitate evidence retrieval and enhance fact-checking by probing information across multiple modalities. Through extensive experiments, we verify if LRQ-FACT can generate relevant FCQs of different types and if LRQ-FACT can consistently outperform baseline methods in multimodal fact-checking. Further analysis illustrates how each component in LRQ-FACT works toward improving the fact-checking performance.

Problem

• LLMs generate relevant FCQs
• Improve multimodal fact-checking
• Enhance evidence retrieval

Innovation

• LLMs generate fact-checking questions
• Multimodal evidence retrieval enhancement
• LRQ-FACT framework outperforms baselines