SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data

2025-04-16

AI Summary

To address the degradation of out-of-distribution (OOD) generalization in fine-tuned pre-trained language models (PLMs) caused by spurious correlations, this paper proposes a structure-aware and large language model (LLM)-guided data augmentation framework. Our method integrates three key components: (1) syntactically informed positive sample generation—guided by part-of-speech and dependency structures—to strengthen modeling of grammatical relations; (2) LLM-generated diverse counterfactual negative samples to explicitly decouple semantic representations from superficial statistical biases; and (3) sentence-level contrastive learning (a SimCLR variant) to enhance representation robustness. Evaluated on cross-domain sentiment classification, gender bias detection, and natural language inference, our approach achieves average OOD accuracy improvements of 3.2–5.7 percentage points over strong baselines, demonstrating substantial gains in both OOD generalization and robustness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Diffusion Models for Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In various natural language processing (NLP) tasks, fine-tuning Pre-trained Language Models (PLMs) often leads to the issue of spurious correlations, which negatively impacts performance, particularly when dealing with out-of-distribution data. To address this problem, we propose SALAD}(Structure Aware and LLM-driven Augmented Data), a novel approach designed to enhance model robustness and generalization by generating structure-aware and counterfactually augmented data for contrastive learning. Our method leverages a tagging-based approach to generate structure-aware positive samples and utilizes large language models (LLMs) to generate counterfactual negative samples with diverse sentence patterns. By applying contrastive learning, SALAD enables the model to focus on learning the structural relationships between key sentence components while minimizing reliance on spurious correlations. We validate our approach through experiments on three tasks: Sentiment Classification, Sexism Detection, and Natural Language Inference. The results demonstrate that SALAD not only improves model robustness and performance across different environments but also enhances generalization to out-of-distribution datasets and cross-domain scenarios.

Problem

• Reducing spurious correlations in fine-tuned PLMs for NLP tasks
• Enhancing model robustness and generalization with contrastive learning
• Generating structure-aware and counterfactual data using LLMs

Innovation

• Structure-aware positive samples via tagging
• LLM-generated counterfactual negative samples
• Contrastive learning for robust generalization