Teaching Language Models to Check Grounded Claim Factuality with Human Test-Taking Strategies

2026-05-28

AI Summary

This work addresses the limitations of existing factuality verification methods, which often rely on dataset-specific thresholds or direct prompting of large language models (LLMs), failing to efficiently elicit their reasoning capabilities. The authors reformulate evidence-based claim verification as a yes/no reading comprehension task and introduce, for the first time, human test-taking strategies to guide LLM reasoning—substantially reducing token consumption. By integrating supervised fine-tuning with a self-correction mechanism, they train compact, interpretable small language models as efficient alternatives to LLMs. The proposed approach achieves state-of-the-art performance on two benchmarks, setting a new record on one, while cutting token usage by over 80% compared to unguided reasoning. The resulting small models maintain competitive accuracy and generate human-interpretable justifications.

Abstract

Grounded claim factuality checking is important for large language model (LLM) applications such as retrieval-augmented generation, as it helps users assess the correctness of generated outputs. Existing metrics using entailment classifiers require dataset-specific threshold tuning, while LLM-based approaches often use direct prompting, which underutilises the reasoning capabilities of LLMs. We address this by formulating grounded claim factuality checking as a true/false reading comprehension task and prompting LLMs with explicit test-taking strategies for efficient reasoning. Our method reduces token usage by over 80% compared to unguided open-ended reasoning, and achieves competitive performance to more expensive alternatives across two factuality benchmarks, setting a new state of the art on one. To further reduce inference cost, we train small language models (SLMs) to replace LLMs in the checking pipeline. Using supervised fine-tuning (SFT) and a self-revision mechanism, the SLMs learn to improve their factuality judgements. Experimental results show that the resulting SLMs perform on par with strong baselines, combining low inference costs with generating supporting rationales to support interpretability. Code and datasets will be released upon acceptance.

Problem

• grounded claim factuality
• large language models
• factuality checking
• reasoning efficiency
• inference cost

Innovation

• grounded factuality checking
• test-taking strategies
• small language models
• self-revision mechanism
• retrieval-augmented generation