Safeguarding System Prompts for LLMs

2024-12-18

AI Summary

To address the vulnerability of system prompts in large language models (LLMs) to leakage—especially via adversarial user queries—leading to unauthorized exposure and misuse of commercially sensitive information, this paper proposes PromptKeeper, a novel defense framework. Methodologically, it formulates prompt leakage detection as a statistical hypothesis testing problem for the first time; introduces a response regeneration mechanism leveraging decoy prompts to preserve output naturalness and interaction consistency under benign conditions; and integrates lightweight runtime detection, side-channel mitigation, and dynamic re-generation strategies. Experimental evaluation across multiple mainstream LLMs demonstrates that PromptKeeper achieves high-accuracy detection of diverse prompt extraction attacks, maintains over 99.2% quality of benign responses (measured by standard linguistic and functional metrics), and incurs less than 3% overhead in inference latency—thereby delivering strong robustness with minimal computational cost.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) are increasingly utilized in applications where system prompts, which guide model outputs, play a crucial role. These prompts often contain business logic and sensitive information, making their protection essential. However, adversarial and even regular user queries can exploit LLM vulnerabilities to expose these hidden prompts. To address this issue, we propose PromptKeeper, a robust defense mechanism designed to safeguard system prompts. PromptKeeper tackles two core challenges: reliably detecting prompt leakage and mitigating side-channel vulnerabilities when leakage occurs. By framing detection as a hypothesis-testing problem, PromptKeeper effectively identifies both explicit and subtle leakage. Upon detection, it regenerates responses using a dummy prompt, ensuring that outputs remain indistinguishable from typical interactions when no leakage is present. PromptKeeper ensures robust protection against prompt extraction attacks via either adversarial or regular queries, while preserving conversational capability and runtime efficiency during benign user interactions.

Problem

• Large Language Models
• Confidentiality
• Prompt Leakage

Innovation

• PromptKeeper
• LLMs Protection
• Detection and Obfuscation