Logic.py: Bridging the Gap between LLMs and Constraint Solvers

2025-02-17

AI Summary

To address the low accuracy of large language models (LLMs) on symbolic reasoning tasks—such as logic puzzles—this paper proposes the LLM-Constraint Hybrid framework. First, an LLM (Llama 3.1 70B) automatically formalizes natural-language puzzle descriptions into Logic.py, a logic-oriented domain-specific language; subsequently, a constraint solver performs exact symbolic execution. This two-stage paradigm enables the first end-to-end, LLM-driven logical modeling and symbolic solving pipeline, overcoming the accuracy limitations of pure LLM-based direct reasoning. Evaluated on the ZebraLogicBench benchmark, our method achieves 90.2% accuracy—surpassing the strongest baseline by 65.1 percentage points and establishing a new state-of-the-art. The results empirically validate the effectiveness and scalability of tightly integrating LLMs with symbolic reasoning tools.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present a novel approach to formalise and solve search-based problems using large language models, which significantly improves upon previous state-of-the-art results. We demonstrate the efficacy of this approach on the logic puzzles benchmark ZebraLogicBench. Instead of letting the LLM attempt to directly solve the puzzles, our method prompts the model to formalise the problem in a logic-focused domain-specific language (DSL) called Logic.py. This formalised representation is then solved using a constraint solver, leveraging the strengths of both the language model and the solver. Our approach achieves a remarkable 65% absolute improvement over the baseline performance of Llama 3.1 70B on ZebraLogicBench, setting a new state-of-the-art with an accuracy of over 90%. This significant advancement demonstrates the potential of combining language models with domain-specific languages and auxiliary tools on traditionally challenging tasks for LLMs.

Problem

• Integrate LLMs with constraint solvers
• Formalize search problems in Logic.py DSL
• Enhance LLM performance on logic puzzles

Innovation

• Combines LLMs with constraint solvers
• Uses Logic.py for problem formalisation
• Achieves 90% accuracy on ZebraLogicBench