LogicGraph : Benchmarking Multi-Path Logical Reasoning via Neuro-Symbolic Generation and Verification

2026-02-24

AI Summary

This work addresses a critical gap in the evaluation of large language models (LLMs), which has predominantly focused on single-path reasoning and thus fails to capture their capabilities in multi-path logical inference. The authors propose LogicGraph, the first benchmark specifically designed for multi-path logical reasoning, leveraging a neuro-symbolic framework that integrates backward logical generation with semantic instantiation to construct high-depth, distractor-rich problems amenable to formal verification. They further introduce a reference-free, multi-dimensional evaluation protocol. Experimental results reveal that mainstream LLMs consistently exhibit premature convergence to a single reasoning path, with their ability to cover multiple valid paths deteriorating significantly as reasoning depth increases—highlighting a fundamental limitation in divergent logical reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Evaluations of large language models (LLMs) primarily emphasize convergent logical reasoning, where success is defined by producing a single correct proof. However, many real-world reasoning problems admit multiple valid derivations, requiring models to explore diverse logical paths rather than committing to one route. To address this limitation, we introduce LogicGraph, the first benchmark aimed to systematically evaluate multi-path logical reasoning, constructed via a neuro-symbolic framework that leverages backward logic generation and semantic instantiation. This pipeline yields solver-verified reasoning problems formalized by high-depth multi-path reasoning and inherent logical distractions, where each instance is associated with an exhaustive set of minimal proofs. We further propose a reference-free evaluation framework to rigorously assess model performance in both convergent and divergent regimes. Experiments on state-of-the-art language models reveal a common limitation: models tend to commit early to a single route and fail to explore alternatives, and the coverage gap grows substantially with reasoning depth. LogicGraph exposes this divergence gap and provides actionable insights to motivate future improvements. Our code and data will be released at https://github.com/kkkkarry/LogicGraph.

Problem

• multi-path logical reasoning
• large language models
• divergent reasoning
• logical proofs
• reasoning evaluation

Innovation

• multi-path logical reasoning
• neuro-symbolic generation
• backward logic generation
• reference-free evaluation
• minimal proof enumeration