MERGE: Minimal Expression-Replacement GEneralization Test for Natural Language Inference

2025-10-28

AI Summary

To address the limited robustness of natural language inference (NLI) models and the high cost and difficulty of manually constructing or automatically generating generalization evaluation benchmarks, this paper proposes MERGE: a method that systematically substitutes open-class lexical items (e.g., nouns, verbs) while preserving the original logical structure and inferential relationships. MERGE employs part-of-speech constraints, frequency-based filtering, and semantic plausibility verification to generate high-quality NLI variants. It establishes the first automatically extensible, inference-structure-preserving generalization evaluation framework. Experiments show that state-of-the-art NLI models suffer 4–20% performance degradation on MERGE variants, revealing pronounced sensitivity to lexical perturbations. Further analysis identifies the semantic categories and occurrence probabilities of open-class words as critical determinants of model robustness.

Technology Category

• Natural Language Processing: Generation
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

In recent years, many generalization benchmarks have shown language models' lack of robustness in natural language inference (NLI). However, manually creating new benchmarks is costly, while automatically generating high-quality ones, even by modifying existing benchmarks, is extremely difficult. In this paper, we propose a methodology for automatically generating high-quality variants of original NLI problems by replacing open-class words, while crucially preserving their underlying reasoning. We dub our generalization test as MERGE (Minimal Expression-Replacements GEneralization), which evaluates the correctness of models' predictions across reasoning-preserving variants of the original problem. Our results show that NLI models' perform 4-20% worse on variants, suggesting low generalizability even on such minimally altered problems. We also analyse how word class of the replacements, word probability, and plausibility influence NLI models' performance.

Problem

• Automatically generates reasoning-preserving NLI problem variants
• Evaluates model robustness through minimal expression replacements
• Analyzes performance degradation across word classes and probabilities

Innovation

• Automatically generates NLI variants by replacing open-class words
• Preserves underlying reasoning in generated test cases
• Evaluates model robustness across reasoning-preserving problem variants