Foundations of Global Consistency Checking with Noisy LLM Oracles

2026-01-20

AI Summary

This work addresses the challenge of verifying global consistency in large-scale natural language fact sets when using large language models (LLMs) as noisy verifiers. It formalizes this problem for the first time and establishes an exponential lower bound on its computational complexity. To tackle this, the authors propose an adaptive divide-and-conquer algorithm that efficiently identifies minimal unsatisfiable subsets (MUSes) and optionally computes minimal repairs based on hitting sets. The approach achieves scalable consistency verification with only a low-order polynomial number of LLM queries. Experimental results demonstrate that the method accurately and efficiently detects and localizes inconsistencies under both synthetic and real-world LLM-induced noise, offering a practical framework for LLM-driven factual consistency validation.

Technology Category

• Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Ensuring that collections of natural-language facts are globally consistent is essential for tasks such as fact-checking, summarization, and knowledge base construction. While Large Language Models (LLMs) can assess the consistency of small subsets of facts, their judgments are noisy, and pairwise checks are insufficient to guarantee global coherence. We formalize this problem and show that verifying global consistency requires exponentially many oracle queries in the worst case. To make the task practical, we propose an adaptive divide-and-conquer algorithm that identifies minimal inconsistent subsets (MUSes) of facts and optionally computes minimal repairs through hitting-sets. Our approach has low-degree polynomial query complexity. Experiments with both synthetic and real LLM oracles show that our method efficiently detects and localizes inconsistencies, offering a scalable framework for linguistic consistency verification with LLM-based evaluators.

Problem

• global consistency
• noisy LLM oracles
• fact consistency
• knowledge base
• inconsistency detection

Innovation

• global consistency
• noisy LLM oracles
• minimal inconsistent subsets
• adaptive divide-and-conquer
• hitting sets