Propositional Logic for Probing Generalization in Neural Networks

2025-06-10

AI Summary

Standard neural architectures—including Transformers, GCNs, and LSTMs—exhibit poor out-of-distribution generalization on propositional logic satisfiability tasks involving novel combinations of logical operators, particularly negation. Method: We construct a de-biased, balanced, and controllable dataset of logical formulas alongside a rigorous evaluation framework to isolate compositional generalization with respect to negation and other operators. Contribution/Results: We provide the first empirical evidence that all models achieve high in-distribution accuracy yet suffer significant performance degradation on negation-based cross-compositional generalization. Critically, only Transformers endowed with explicit structural inductive biases—e.g., syntactic tree encodings or logical structure awareness—demonstrate systematic, compositional application of negation. These findings establish structural priors as a necessary condition for logical generalization, challenging the assumption that purely sequential modeling suffices for symbolic reasoning. Our work highlights the insufficiency of standard architectural choices for robust logical inference and underscores the need for architecture-aware design in neuro-symbolic learning.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Machine Learning: Neuro-Symbolic Learning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The extent to which neural networks are able to acquire and represent symbolic rules remains a key topic of research and debate. Much current work focuses on the impressive capabilities of large language models, as well as their often ill-understood failures on a wide range of reasoning tasks. In this paper, in contrast, we investigate the generalization behavior of three key neural architectures (Transformers, Graph Convolution Networks and LSTMs) in a controlled task rooted in propositional logic. The task requires models to generate satisfying assignments for logical formulas, making it a structured and interpretable setting for studying compositionality. We introduce a balanced extension of an existing dataset to eliminate superficial patterns and enable testing on unseen operator combinations. Using this dataset, we evaluate the ability of the three architectures to generalize beyond the training distribution. While all models perform well in-distribution, we find that generalization to unseen patterns, particularly those involving negation, remains a significant challenge. Transformers fail to apply negation compositionally, unless structural biases are introduced. Our findings highlight persistent limitations in the ability of standard architectures to learn systematic representations of logical operators, suggesting the need for stronger inductive biases to support robust rule-based reasoning.

Problem

• Study neural networks' ability to generalize symbolic rules
• Evaluate generalization on unseen logical operator combinations
• Assess compositional reasoning challenges, especially with negation

Innovation

• Balanced dataset extension for unbiased testing
• Three architectures tested on logical generalization
• Structural biases improve Transformers' negation handling