Althea: Human-AI Collaboration for Fact-Checking and Critical Reasoning

2025-12-29

arXiv.org

AI Summary

Current fact-checking systems struggle to simultaneously achieve scalability, transparency, and the sustained cultivation of users’ critical reasoning skills. This work proposes a retrieval-augmented framework that integrates question generation, evidence retrieval, and structured reasoning chains, introducing— for the first time—the pedagogical concept of instructional scaffolding into human-AI collaborative fact-checking. By balancing guided and self-directed multimodal interactions, the system reconciles immediate verification accuracy with long-term cognitive internalization. Evaluated on the AVeriTeC benchmark, it achieves a Macro-F1 score of 0.44, significantly outperforming baseline methods. A longitudinal study with 963 participants demonstrates that guided interaction enhances immediate judgment accuracy, whereas autonomous exploration fosters more durable cognitive gains.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Humans and AI: Learning Human Values and Preferences
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The web's information ecosystem demands fact-checking systems that are both scalable and epistemically trustworthy. Automated approaches offer efficiency but often lack transparency, while human verification remains slow and inconsistent. We introduce Althea, a retrieval-augmented system that integrates question generation, evidence retrieval, and structured reasoning to support user-driven evaluation of online claims. On the AVeriTeC benchmark, Althea achieves a Macro-F1 of 0.44, outperforming standard verification pipelines and improving discrimination between supported and refuted claims. We further evaluate Althea through a controlled user study and a longitudinal survey experiment (N = 642), comparing three interaction modes that vary in the degree of scaffolding: an Exploratory mode with guided reasoning, a Summary mode providing synthesized verdicts, and a Self-search mode that offers procedural guidance without algorithmic intervention. Results show that guided interaction produces the strongest immediate gains in accuracy and confidence, while self-directed search yields the most persistent improvements over time. This pattern suggests that performance gains are not driven solely by effort or exposure, but by how cognitive work is structured and internalized.

Problem

• fact-checking
• human-AI collaboration
• critical reasoning
• epistemic trust
• information verification

Innovation

• retrieval-augmented reasoning
• human-AI collaboration
• fact-checking
• cognitive scaffolding
• critical reasoning