Learning What to Fail On: Failure-Mode Contextual Bandits for Adversarial Data Curation

2026-08-19

AI Summary

该研究通过将对抗性数据整理视为失败模式的上下文强盗问题，利用检索增强提示生成候选示例，并采用基于验证的奖励策略选择失败模式进行再训练，以提高自然语言理解的鲁棒性。

Abstract

We introduce a failure-aware adversarial retrieval-augmented framework for improving robustness in natural language understanding. Rather than selecting synthetic examples with a fixed reward threshold, our method formulates adversarial data curation as a failure-mode contextual bandit problem. Candidate examples are generated with retrieval-augmented prompting, filtered by the current target model, automatically validated by an LLM judge ensemble, and clustered into recurring failure modes. A stochastic policy then selects which failure modes to sample for retraining, and is updated using validation-based reward that balances robustness gains, forgetting, and data cost. This makes the data curator itself the learning agent, enabling adaptive selection of the most useful model failures across training rounds. On standard benchmarks, our approach improves RoBERTa-base accuracy from 88.48% to 92.60% on SNLI, from 75.04% to 80.95% on ANLI, and from 54.67% to 71.99% on MultiNLI, while consistently outperforming prior adversarial augmentation methods. We further demonstrate transfer to FEVER fact verification, achieving up to 79.86\% FEVER score and 82.45\% accuracy with RoBERTa-large. Finally, we provide a theoretical interpretation showing that, under stated assumptions, failure-mode sampling can reduce shortcut-aligned gradient contributions while inducing bounded distributional drift. By combining retrieval, automated validation, contextual-bandit failure selection, and controlled adversarial retraining, our framework enables scalable robustness improvement without additional human annotation.

Problem

• adversarial data curation
• natural language understanding
• failure modes
• robustness improvement
• contextual bandit

Innovation

• failure-mode contextual bandit
• adversarial data curation
• retrieval-augmented prompting
• automated validation by LLM judge ensemble
• robustness improvement