Data Poisoning for In-context Learning

2024-02-03

North American Chapter of the Association for Computational Linguistics

AI Summary

This work identifies, for the first time, a critical vulnerability of in-context learning (ICL) in large language models (LLMs) to data poisoning attacks. Exploiting ICL’s reliance on exemplars without parameter fine-tuning, we propose ICLPoison—a novel attack framework that applies discrete textual perturbations to context examples to manipulate model internal representations and systematically degrade task performance. We design three task-agnostic, transferable attack strategies: semantic confusion, structural disruption, and distributional shift. Extensive evaluation across multiple LLMs—including GPT-4 and Llama—and diverse NLP benchmarks demonstrates consistent accuracy drops of 20–50%, revealing a fundamental security flaw inherent to ICL mechanisms. Our findings establish the first systematic evidence of ICL’s susceptibility to input-level poisoning, lay the foundation for robustness research on ICL, and underscore urgent concerns regarding data integrity and trustworthiness in real-world LLM deployments.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

In the domain of large language models (LLMs), in-context learning (ICL) has been recognized for its innovative ability to adapt to new tasks, relying on examples rather than retraining or fine-tuning. This paper delves into the critical issue of ICL's susceptibility to data poisoning attacks, an area not yet fully explored. We wonder whether ICL is vulnerable, with adversaries capable of manipulating example data to degrade model performance. To address this, we introduce ICLPoison, a specialized attacking framework conceived to exploit the learning mechanisms of ICL. Our approach uniquely employs discrete text perturbations to strategically influence the hidden states of LLMs during the ICL process. We outline three representative strategies to implement attacks under our framework, each rigorously evaluated across a variety of models and tasks. Our comprehensive tests, including trials on the sophisticated GPT-4 model, demonstrate that ICL's performance is significantly compromised under our framework. These revelations indicate an urgent need for enhanced defense mechanisms to safeguard the integrity and reliability of LLMs in applications relying on in-context learning.

Problem

• Investigates ICL vulnerability to data poisoning attacks
• Proposes ICLPoison framework to exploit ICL learning mechanisms
• Demonstrates significant performance degradation in LLMs via attacks

Innovation

• ICLPoison framework exploits ICL vulnerabilities
• Discrete text perturbations manipulate hidden states
• Three attack strategies evaluated across models