DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Models

2025-09-19

AI Summary

Existing logical reasoning benchmarks suffer from insufficient linguistic diversity and distributional bias, leading to distorted model evaluations. To address this, we propose DivLogicEval—the first debiased evaluation framework for classical logical reasoning. Our approach (1) generates counterintuitive, linguistically diverse natural-language instances grounded in formal logic, mitigating skill entanglement; (2) introduces a decoupled evaluation metric that explicitly controls for logical dependency strength; and (3) validates effectiveness via controlled-variable experiments. Empirical results demonstrate that DivLogicEval significantly discriminates among state-of-the-art large language models in terms of pure logical reasoning capability, uncovering persistent weaknesses under semantically neutral conditions. By improving linguistic coverage and reducing confounding biases, DivLogicEval establishes a more reliable, comprehensive benchmark for assessing foundational logical reasoning competence.

Technology Category

• Knowledge Representation and Reasoning: Logic Programming
• Machine Learning: Statistical Relational/Logic Learning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Logic reasoning in natural language has been recognized as an important measure of human intelligence for Large Language Models (LLMs). Popular benchmarks may entangle multiple reasoning skills and thus provide unfaithful evaluations on the logic reasoning skill. Meanwhile, existing logic reasoning benchmarks are limited in language diversity and their distributions are deviated from the distribution of an ideal logic reasoning benchmark, which may lead to biased evaluation results. This paper thereby proposes a new classical logic benchmark DivLogicEval, consisting of natural sentences composed of diverse statements in a counterintuitive way. To ensure a more reliable evaluation, we also introduce a new evaluation metric that mitigates the influence of bias and randomness inherent in LLMs. Through experiments, we demonstrate the extent to which logical reasoning is required to answer the questions in DivLogicEval and compare the performance of different popular LLMs in conducting logical reasoning.

Problem

• Evaluating logical reasoning in LLMs with diverse natural language
• Addressing bias from limited language diversity in benchmarks
• Introducing a new metric to reduce bias and randomness

Innovation

• Classical logic benchmark with counterintuitive statements
• New evaluation metric reducing bias and randomness
• Diverse natural language sentences for reliable testing