How to Improve the Robustness of Closed-Source Models on NLI

2025-05-26

AI Summary

Closed-source large language models (LLMs) exhibit insufficient robustness and limited controllability on out-of-distribution (OOD) data in natural language inference (NLI) tasks. Method: We propose a parameter-agnostic, training-free, data-centric augmentation strategy that requires no access to model internals or retraining. Contribution/Results: First, we systematically identify OOD data complexity as a decisive factor governing robustness improvement. Second, we empirically establish that closed-source autoregressive LLMs inherently outperform traditional encoder-based models on OOD NLI, positioning them as a new robustness benchmark. Third, we design an augmentation method combining challenge-sample resampling with LLM-generated data substitution, and introduce a standardized OOD evaluation framework. Experiments demonstrate robustness improvements of 1.5% on high-complexity OOD data and 3.7% on low-complexity OOD data, validating both the efficacy of our approach and the utility of the proposed benchmark.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Closed-source Large Language Models (LLMs) have become increasingly popular, with impressive performance across a wide range of natural language tasks. These models can be fine-tuned to further improve performance, but this often results in the models learning from dataset-specific heuristics that reduce their robustness on out-of-distribution (OOD) data. Existing methods to improve robustness either perform poorly, or are non-applicable to closed-source models because they assume access to model internals, or the ability to change the model's training procedure. In this work, we investigate strategies to improve the robustness of closed-source LLMs through data-centric methods that do not require access to model internals. We find that the optimal strategy depends on the complexity of the OOD data. For highly complex OOD datasets, upsampling more challenging training examples can improve robustness by up to 1.5%. For less complex OOD datasets, replacing a portion of the training set with LLM-generated examples can improve robustness by 3.7%. More broadly, we find that large-scale closed-source autoregressive LLMs are substantially more robust than commonly used encoder models, and are a more appropriate choice of baseline going forward.

Problem

• Enhancing robustness of closed-source LLMs on OOD data
• Overcoming limitations of existing methods for closed-source models
• Identifying optimal data-centric strategies for different OOD complexities

Innovation

• Data-centric methods improve closed-source LLM robustness
• Upsampling challenging examples boosts complex OOD performance
• LLM-generated examples enhance simpler OOD robustness