When Verification Fails: How Compositionally Infeasible Claims Escape Rejection

2026-04-13

AI Summary

Current benchmarks for verifying scientific claims struggle to distinguish whether models perform full constraint validation or merely rely on salient constraints, often misclassifying compositionally infeasible claims—those violating structural consistency despite satisfying salient constraints—as true. This work introduces the first method for constructing compositionally infeasible claims, integrating a closed-world assumption framework, contextual intervention, and ROC analysis to systematically reveal across multimodal settings and multiple model families that contemporary models predominantly exploit salient-constraint shortcuts. The study demonstrates that this verification bottleneck stems from structural threshold biases rather than insufficient reasoning capacity and cannot be mitigated by prompting strategies. The proposed approach effectively differentiates rigorous validation from superficial heuristics, exposing fundamental limitations in current claim-verification systems.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Scientific claim verification, the task of determining whether claims are entailed by scientific evidence, is fundamental to establishing discoveries in evidence while preventing misinformation. This process involves evaluating each asserted constraint against validated evidence. Under the Closed-World Assumption (CWA), a claim is accepted if and only if all asserted constraints are positively supported. We show that existing verification benchmarks cannot distinguish models enforcing this standard from models applying a simpler shortcut called salient-constraint checking, which applies CWA's rejection criterion only to the most salient constraint and accepts when that constraint is supported. Because existing benchmarks construct infeasible claims by perturbing a single salient element they are insufficient at distinguishing between rigorous claim verification and simple salient-constraint reliance. To separate the two, we construct compositionally infeasible claims where the salient constraint is supported but a non-salient constraint is contradicted. Across model families and modalities, models that otherwise saturate existing benchmarks consistently over-accept these claims, confirming the prevalence of such shortcut reasoning. Via model context interventions, we show that different models and prompting strategies occupy distinct positions on a shared ROC curve, indicating that the gap between model families reflects differences in verification threshold rather than underlying reasoning ability, and that the compositional inference bottleneck is a structural property of current verification behavior that strategy guidance alone cannot overcome.

Problem

• scientific claim verification
• Closed-World Assumption
• compositional infeasibility
• salient-constraint checking
• verification benchmarks

Innovation

• scientific claim verification
• compositional infeasibility
• salient-constraint checking
• Closed-World Assumption
• verification bottleneck