System Prompt Poisoning: Persistent Attacks on Large Language Models Beyond User Injection

2025-05-10

AI Summary

This work identifies and systematically investigates *System Prompt Poisoning*—a novel LLM security threat wherein adversaries persistently hijack model behavior by tampering with the system prompt, thereby compromising all subsequent user interactions and transcending conventional input-based injection attacks. We formally define this attack and propose four practical poisoning strategies. We validate them across diverse tasks—including mathematical reasoning, programming, logical deduction, and NLP—and across advanced techniques such as chain-of-thought (CoT) and retrieval-augmented generation (RAG), using both open- and closed-source models (e.g., Llama, GPT). Experiments demonstrate high attack success rates without requiring jailbreaking, and critically reveal severe performance degradation in CoT and RAG pipelines under poisoning. This study fills a critical gap in system-level LLM security research and establishes foundational methodology for robustness evaluation and defense of system prompts.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Large language models (LLMs) have gained widespread adoption across diverse applications due to their impressive generative capabilities. Their plug-and-play nature enables both developers and end users to interact with these models through simple prompts. However, as LLMs become more integrated into various systems in diverse domains, concerns around their security are growing. Existing studies mainly focus on threats arising from user prompts (e.g. prompt injection attack) and model output (e.g. model inversion attack), while the security of system prompts remains largely overlooked. This work bridges the critical gap. We introduce system prompt poisoning, a new attack vector against LLMs that, unlike traditional user prompt injection, poisons system prompts hence persistently impacts all subsequent user interactions and model responses. We systematically investigate four practical attack strategies in various poisoning scenarios. Through demonstration on both generative and reasoning LLMs, we show that system prompt poisoning is highly feasible without requiring jailbreak techniques, and effective across a wide range of tasks, including those in mathematics, coding, logical reasoning, and natural language processing. Importantly, our findings reveal that the attack remains effective even when user prompts employ advanced prompting techniques like chain-of-thought (CoT). We also show that such techniques, including CoT and retrieval-augmentation-generation (RAG), which are proven to be effective for improving LLM performance in a wide range of tasks, are significantly weakened in their effectiveness by system prompt poisoning.

Problem

• Investigates system prompt poisoning attacks on LLMs
• Explores persistent impact on user interactions and responses
• Assesses weakening of advanced prompting techniques by poisoning

Innovation

• Introduces system prompt poisoning attack
• Investigates four practical attack strategies
• Shows effectiveness across diverse LLM tasks