Generics and Default Reasoning in Large Language Models

2025-08-19

AI Summary

This study investigates the capacity of large language models (LLMs) to perform defeasible reasoning—a core aspect of nonmonotonic logic—across 20 patterns (e.g., “birds fly,” “crows are black”), focusing on their understanding of generic generalizations (non-universal quantifications), exception tolerance, and default inference. Method: We systematically evaluate 28 LLMs using zero-shot, few-shot, and chain-of-thought (CoT) prompting on canonical default reasoning tasks. Contribution/Results: We find that LLMs consistently conflate defeasible with deductive reasoning, misinterpreting generic statements as universal logical quantifications. Surprisingly, CoT prompting significantly degrades performance, revealing an absence of structured default inference mechanisms. This work provides the first rigorous diagnosis of fundamental limitations in current LLMs’ nonmonotonic reasoning capabilities, establishing a novel benchmark and critical insights for developing next-generation models with robust commonsense default reasoning.

Technology Category

• Knowledge Representation and Reasoning: Nonmonotonic Reasoning
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper evaluates the capabilities of 28 large language models (LLMs) to reason with 20 defeasible reasoning patterns involving generic generalizations (e.g., 'Birds fly', 'Ravens are black') central to non-monotonic logic. Generics are of special interest to linguists, philosophers, logicians, and cognitive scientists because of their complex exception-permitting behaviour and their centrality to default reasoning, cognition, and concept acquisition. We find that while several frontier models handle many default reasoning problems well, performance varies widely across models and prompting styles. Few-shot prompting modestly improves performance for some models, but chain-of-thought (CoT) prompting often leads to serious performance degradation (mean accuracy drop -11.14%, SD 15.74% in models performing above 75% accuracy in zero-shot condition, temperature 0). Most models either struggle to distinguish between defeasible and deductive inference or misinterpret generics as universal statements. These findings underscore both the promise and limits of current LLMs for default reasoning.

Problem

• Evaluating LLMs' default reasoning with generics
• Assessing performance across models and prompting styles
• Identifying struggles with defeasible vs deductive inference

Innovation

• Evaluated 28 LLMs on defeasible reasoning patterns
• Tested few-shot and chain-of-thought prompting techniques
• Analyzed models' ability to distinguish deductive vs defeasible inference