Exploring Multidimensional Checkworthiness: Designing AI-assisted Claim Prioritization for Human Fact-checkers

2024-12-11

arXiv.org

AI Summary

With the proliferation of online misinformation, limited human fact-checking resources necessitate scientifically grounded prioritization mechanisms. Method: We propose a multidimensional verifiability framework that models claim prioritization as an information retrieval task integrating subjective judgment, contextual constraints, and implicit hierarchical strategies—departing from conventional single-metric ranking paradigms—and design an LLM-augmented AI-assisted prototype enabling dynamic, personalized strategy adaptation by fact-checkers. Contribution/Results: Through Research through Design, mixed-methods evaluation, and empirical study with 16 professional fact-checkers, we first identify and formally characterize implicit hierarchical strategies in fact-checking practice and distill actionable design principles. Results demonstrate significant improvements in verification efficiency and subjective strategy alignment, empirically validating the effectiveness and practical utility of multidimensional ranking within real-world fact-checking workflows.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Preferences

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Given the massive volume of potentially false claims circulating online, claim prioritization is essential in allocating limited human resources available for fact-checking. In this study, we perceive claim prioritization as an information retrieval (IR) task: just as multidimensional IR relevance, with many factors influencing which search results a user deems relevant, checkworthiness is also multi-faceted, subjective, and even personal, with many factors influencing how fact-checkers triage and select which claims to check. Our study investigated both the multidimensional nature of checkworthiness and effective tool support to assist fact-checkers in claim prioritization. Methodologically, we pursued Research through Design combined with mixed-method evaluation. Specifically, we developed an AI-assisted claim prioritization prototype as a probe to explore how fact-checkers use multidimensional checkworthy factors to prioritize claims, simultaneously probing fact-checker needs and exploring the design space to meet those needs. With 16 professional fact-checkers participating in our study, we uncovered a hierarchical prioritization strategy fact-checkers implicitly use, revealing an underexplored aspect of their workflow, with actionable design recommendations for improving claim triage across multidimensional checkworthiness and tailoring this process with LLM integration.

Problem

• Designing AI tools for prioritizing false claims in fact-checking
• Exploring multidimensional factors influencing claim checkworthiness
• Improving human fact-checker workflow with LLM integration

Innovation

• AI-assisted claim prioritization prototype
• multidimensional checkworthy factors analysis
• LLM integration for workflow tailoring