Normative Reasoning in Large Language Models: A Comparative Benchmark from Logical and Modal Perspectives

2025-10-30

AI Summary

Large language models (LLMs) exhibit uncharacterized inconsistencies and cognitive biases in normative reasoning (e.g., obligation, permission) and epistemic reasoning (e.g., knowledge, belief), limiting their reliability in ethically or legally sensitive applications. Method: We introduce the first benchmark integrating formal logical verification with cognitive variable modeling, featuring a high-quality, manually curated dataset covering both deontic and epistemic modalities. Contribution/Results: Empirical evaluation reveals that while LLMs perform basic valid inferences, they suffer from systematic inconsistency in normative tasks—exhibiting human-like biases such as permission misinterpretation and overgeneralization of obligations. Crucially, these bias patterns closely align with findings from empirical cognitive science. Our work uncovers fundamental limitations in current LLMs’ capacity to model deontic logic accurately, providing both empirical grounding and open-source evaluation tools (code and data) for normative alignment and trustworthy AI development.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Normative reasoning is a type of reasoning that involves normative or deontic modality, such as obligation and permission. While large language models (LLMs) have demonstrated remarkable performance across various reasoning tasks, their ability to handle normative reasoning remains underexplored. In this paper, we systematically evaluate LLMs' reasoning capabilities in the normative domain from both logical and modal perspectives. Specifically, to assess how well LLMs reason with normative modals, we make a comparison between their reasoning with normative modals and their reasoning with epistemic modals, which share a common formal structure. To this end, we introduce a new dataset covering a wide range of formal patterns of reasoning in both normative and epistemic domains, while also incorporating non-formal cognitive factors that influence human reasoning. Our results indicate that, although LLMs generally adhere to valid reasoning patterns, they exhibit notable inconsistencies in specific types of normative reasoning and display cognitive biases similar to those observed in psychological studies of human reasoning. These findings highlight challenges in achieving logical consistency in LLMs' normative reasoning and provide insights for enhancing their reliability. All data and code are released publicly at https://github.com/kmineshima/NeuBAROCO.

Problem

• Evaluating LLMs' normative reasoning from logical and modal perspectives
• Comparing reasoning with normative and epistemic modals using new dataset
• Identifying inconsistencies and cognitive biases in LLMs' normative reasoning

Innovation

• Systematically evaluated LLMs' normative reasoning capabilities
• Introduced new dataset with normative and epistemic reasoning patterns
• Compared reasoning with normative modals versus epistemic modals