Problem
Research questions and friction points this paper is trying to address.
compositional reasoning
logical operators
compound answer options
large language models
atomic judgments
From Atomic Evidence to Logical Composition: Structured Compositional Reasoning over Compound Answer Options
🤖 AI Summary
Large language models exhibit poor performance on compound judgments composed of logical operators such as AND, OR, and NEITHER/NOR, even when they accurately handle atomic judgments. This work proposes the first framework that reasons about such compounds without requiring explicit composite inputs, relying solely on atomic evidence and logical constraints. The approach decomposes compound questions into atomic subproblems, generates and scores contrastive hypotheses, and then integrates these scores via integer linear programming subject to logical operator constraints for calibrated fusion. This method substantially improves model comprehension of complex logical structures—particularly challenging cases involving NEITHER/NOR—achieving Macro-F1 gains from 48.3 to 77.0 on LOGICAL-COMMONSENSEQA and from 47.0 to 75.6 on the newly introduced LOGICAL-SATA benchmark.
📝 Abstract
Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.
Innovation
Methods, ideas, or system contributions that make the work stand out.
structured compositional reasoning
atomic decomposition
contrastive hypothesis scoring
operator-constrained ILP
logical composition
🔎 Similar Papers
No similar papers found.
