Understanding and Mitigating Prompt Leaking Attacks in Real-World LLM-Based Applications

2026-06-17

AI Summary

This study addresses the widespread risk of prompt leakage in real-world large language model (LLM) applications, which often exposes system prompts and sensitive information such as API keys. The authors present the first systematic evaluation of 1,200 LLM applications across six commercial platforms, demonstrating the prevalence of such vulnerabilities and introducing an “attention drift” mechanism to explain why existing defenses fail. Building on this insight, they propose AREA, a lightweight defense that leverages attention re-anchoring to achieve state-of-the-art leakage prevention while improving average usability by over 33% and reducing optimization overhead by nearly threefold. Their findings have already prompted two vendors to classify the identified vulnerability as medium-severity.

Abstract

Large language model (LLM)-based applications rely on system prompts to encode core logic and developer-defined constraints, making these prompts important intellectual property. However, system prompts are vulnerable to prompt leaking attacks. Although prior work has shown such attacks in controlled settings, their prevalence, causes, and defenses in real-world deployments remain unclear. This paper presents a systematic study of prompt leaking in real-world LLM-based applications. We measure 1,200 applications across six major commercial platforms and find that over 80% of deployments leak system prompts under realistic adversarial queries, sometimes exposing sensitive information such as third-party API keys. We also show that existing defenses often fail to prevent leakage without degrading usability. To explain these failures, we conduct an attention-level mechanistic analysis and identify attention drift, where query-key alignment bias and softmax amplification cause LLMs to progressively ignore defensive constraints. Guided by this insight, we propose AREA, a practical defense that re-anchors the model's attention using an optimizable soft prompt. Experiments and real-world case studies show that AREA matches the leakage resistance of state-of-the-art defenses while improving average usability by over 33% and reducing optimization overhead by nearly 3x. Our responsible disclosure led two affected vendors to classify these leaks as medium-severity vulnerabilities.

Problem

• prompt leaking
• LLM security
• system prompt
• adversarial attacks
• intellectual property

Innovation

• prompt leaking
• attention drift
• soft prompt
• LLM security
• AREA defense