Thunder-NUBench: A Benchmark for LLMs' Sentence-Level Negation Understanding

2025-06-17

AI Summary

Existing benchmarks treat negation as a peripheral phenomenon in tasks like natural language inference, lacking dedicated evaluation for sentence-level negation understanding. Method: We introduce NegBench—the first structured benchmark explicitly designed to assess large language models’ (LLMs) sentence-level negation comprehension—covering semantic variants including standard negation, local negation, contradiction, and paraphrase. It employs human-annotated sentence pairs and multiple-choice discrimination tasks for fine-grained, semantically grounded evaluation. Contribution/Results: NegBench is the first framework to isolate negation understanding as a primary objective—not a byproduct—and introduces structured negative exemplar contrast. It provides a high-quality, reproducible, and interpretable quantitative evaluation framework. Experiments reveal systematic failures of mainstream LLMs in negation logic reasoning, establishing NegBench as a critical diagnostic and improvement tool for negation-aware model development.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Negation is a fundamental linguistic phenomenon that poses persistent challenges for Large Language Models (LLMs), particularly in tasks requiring deep semantic understanding. Existing benchmarks often treat negation as a side case within broader tasks like natural language inference, resulting in a lack of benchmarks that exclusively target negation understanding. In this work, we introduce extbf{Thunder-NUBench}, a novel benchmark explicitly designed to assess sentence-level negation understanding in LLMs. Thunder-NUBench goes beyond surface-level cue detection by contrasting standard negation with structurally diverse alternatives such as local negation, contradiction, and paraphrase. The benchmark consists of manually curated sentence-negation pairs and a multiple-choice dataset that enables in-depth evaluation of models' negation understanding.

Problem

• Assessing LLMs' sentence-level negation understanding
• Addressing lack of negation-focused benchmarks
• Evaluating deep semantic negation comprehension

Innovation

• Introduces Thunder-NUBench for negation understanding
• Contrasts standard negation with diverse alternatives
• Uses manually curated sentence-negation pairs