Making Language Models Robust Against Negation

2025-02-11

AI Summary

Language models exhibit persistent robustness deficiencies in negation understanding. To address this, we propose a self-supervised negation robustness enhancement method centered on a novel pretraining task—Next Sentence Polarity Prediction (NSPP)—which explicitly incorporates negation polarity modeling into the sentence relation prediction framework without requiring manual annotation. As a semantically enriched variant of Next Sentence Prediction (NSP), NSPP is jointly optimized with standard masked language modeling (MLM), ensuring full compatibility with BERT and RoBERTa architectures. Evaluated across nine negation understanding benchmarks, our approach consistently outperforms strong baselines: on the CondaQA question answering task, it achieves F1 improvements of 1.8–9.1%, demonstrating enhanced deep logical reasoning over negation. This work establishes a scalable, self-supervised paradigm for robust negation understanding—particularly valuable in low-resource settings—while preserving architectural generality and training efficiency.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Neuro-Symbolic Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Negation has been a long-standing challenge for language models. Previous studies have shown that they struggle with negation in many natural language understanding tasks. In this work, we propose a self-supervised method to make language models more robust against negation. We introduce a novel task, Next Sentence Polarity Prediction (NSPP), and a variation of the Next Sentence Prediction (NSP) task. We show that BERT and RoBERTa further pre-trained on our tasks outperform the off-the-shelf versions on nine negation-related benchmarks. Most notably, our pre-training tasks yield between 1.8% and 9.1% improvement on CondaQA, a large question-answering corpus requiring reasoning over negation.

Problem

• Improve language models' negation handling
• Develop self-supervised negation robustness method
• Enhance performance on negation-related benchmarks

Innovation

• Self-supervised negation robustness method
• Next Sentence Polarity Prediction task
• Enhanced BERT and RoBERTa performance