Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability

2026-05-27

AI Summary

Existing evaluation methods for assessing large language models’ reasoning capabilities on Boolean satisfiability (SAT) problems are susceptible to prediction biases, often failing to reflect genuine logical reasoning performance. This work proposes a novel evaluation protocol based on minimally different paired instances and introduces the Accurate Distinction Rate (ADR) metric to systematically assess model performance on 2-SAT, 3-SAT, and their classic reductions—vertex cover and discrete 3D bin packing. Experimental results reveal that most models exhibit inflated performance under conventional metrics yet fail to reproduce the characteristic phase transition behavior of 3-SAT. In contrast, ADR effectively identifies models with authentic reasoning abilities, and over 80% of instances demonstrate consistent decisions across different problem representations, confirming the reliability and cross-representational consistency of the proposed approach.

Abstract

Large language models (LLMs) are increasingly used for tasks that implicitly reduce to Boolean satisfiability (SAT), yet their reasoning ability on SAT remains unclear. We present a systematic study of LLMs on 2-SAT and 3-SAT, together with two canonical reductions, Vertex Cover and discrete 3D packing, to probe representation-invariant reasoning. We first evaluate models using conventional metrics, including accuracy, precision, recall, and F1, as well as the SAT phase-transition setting. We find that these metrics can be misleading: many models obtain high scores by over-predicting satisfiable formulas, fail to reproduce the classical easy-hard-easy signature around the 3-SAT threshold, and degrade sharply as the number of variables grows. To address this problem, we introduce a paired-formula protocol based on minimally different satisfiable and unsatisfiable instances, together with Accurate Differentiation Rate (ADR), which requires both members of each pair to be classified correctly. ADR separates reasoning-oriented models from heuristic ones and correlates with witness validity. Beyond CNF, we test cross-representation consistency by converting CNF to Vertex Cover and 3-SAT to discrete 3D packing. Model decisions on CNF and on the corresponding graph or packing instances agree for most models on more than 80 percent of instances, suggesting stable decision rules across representations. Overall, our results show that SAT is a conservative probe for LLM reasoning, and that paired evaluation with ADR provides a more faithful and representation-robust assessment than conventional metrics.

Problem

• Satisfiability Solving
• Large Language Models
• Reasoning Capability
• Evaluation Metrics
• Boolean Satisfiability

Innovation

• paired-formula evaluation
• Accurate Differentiation Rate (ADR)
• representation-invariant reasoning
• SAT solving with LLMs
• cross-representation consistency