A Theorem-Proving-Based Evaluation of Neural Semantic Parsing

2025-10-13

AI Summary

Existing neural semantic parsers are predominantly evaluated using graph-matching metrics (e.g., Smatch), which fail to guarantee logical equivalence. This work proposes a logic-sensitive evaluation framework that jointly employs graph matching and first-order logic theorem proving—specifically, bidirectional entailment checking—to rigorously assess semantic equivalence. Experiments across T5 and GPT-family models, under both supervised fine-tuning and in-context learning settings, quantitatively demonstrate that high Smatch scores do not ensure logical correctness. While normalized target representations significantly improve formula well-formedness and logical sufficiency, models consistently err on critical logical aspects—including variable binding and predicate naming. The study reveals severe limitations of conventional metrics in capturing logical fidelity and underscores the necessity of formal logical equivalence verification. It establishes a new evaluation benchmark for semantic parsing and provides concrete directions for improving assessment rigor and model reliability.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Common-Sense Reasoning
• Machine Learning: Statistical Relational/Logic Learning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Graph-matching metrics such as Smatch are the de facto standard for evaluating neural semantic parsers, yet they capture surface overlap rather than logical equivalence. We reassess evaluation by pairing graph-matching with automated theorem proving. We compare two approaches to building parsers: supervised fine-tuning (T5-Small/Base) and few-shot in-context learning (GPT-4o/4.1/5), under normalized and unnormalized targets. We evaluate outputs using graph-matching, bidirectional entailment between source and target formulas with a first-order logic theorem prover, and well-formedness. Across settings, we find that models performing well on graph-matching often fail to produce logically equivalent formulas. Normalization reduces incidental target variability, improves well-formedness, and strengthens logical adequacy. Error analysis shows performance degrades with increasing formula complexity and with coordination, prepositional phrases, and passive voice; the dominant failures involve variable binding and indexing, and predicate naming. These findings highlight limits of graph-based metrics for reasoning-oriented applications and motivate logic-sensitive evaluation and training objectives together with simplified, normalized target representations. All code and data for our experiments are publicly available.

Problem

• Evaluating neural semantic parsers beyond surface-level graph matching metrics
• Assessing logical equivalence of semantic parses using automated theorem proving
• Identifying limitations of current evaluation methods for reasoning applications

Innovation

• Combining graph-matching with automated theorem proving
• Evaluating bidirectional entailment using logic theorem prover
• Using normalized targets to improve logical adequacy