🤖 AI Summary
This study addresses two critical adversarial threats to large language models (LLMs): prompt injection and goal hijacking. We systematically evaluate adversarial fine-tuning as a defense mechanism—first demonstrating its efficacy against such attacks—by constructing a comprehensive benchmark of diverse adversarial examples and empirically assessing GPT-3 variants alongside contemporaneous state-of-the-art LLMs. Results show that, without mitigation, attack success rates reach 31%; after adversarial fine-tuning, the success rate drops to near zero for smaller GPT-3 models, confirming the method’s strong defensive capability. Crucially, we uncover a robust size–security trade-off: larger models (e.g., Davinci) exhibit significantly higher vulnerability. Our work establishes a reproducible defense paradigm for enhancing LLM robustness and provides foundational empirical evidence for adversarial resilience in modern language models.
📝 Abstract
This paper documents early research conducted in 2022 on defending against prompt injection attacks in large language models, providing historical context for the evolution of this critical security domain. This research focuses on two adversarial attacks against Large Language Models (LLMs): prompt injection and goal hijacking. We examine how to construct these attacks, test them on various LLMs, and compare their effectiveness. We propose and evaluate a novel defense technique called Adversarial Fine-Tuning. Our results show that, without this defense, the attacks succeeded 31% of the time on GPT-3 series models. When using our Adversarial Fine-Tuning approach, attack success rates were reduced to near zero for smaller GPT-3 variants (Ada, Babbage, Curie), though we note that subsequent research has revealed limitations of fine-tuning-based defenses. We also find that more flexible models exhibit greater vulnerability to these attacks. Consequently, large models such as GPT-3 Davinci are more vulnerable than smaller models like GPT-2. While the specific models tested are now superseded, the core methodology and empirical findings contributed to the foundation of modern prompt injection defense research, including instruction hierarchy systems and constitutional AI approaches.