Inadvertent Context Leakage in Language Models

2026-08-20

AI Summary

研究探讨了语言模型在处理敏感用户信息时的无意上下文泄露问题，并通过一种新的自适应攻击方法利用这种泄露，展示了即使在非对抗性请求下也能高精度地重构秘密。

Abstract

For AI agents to be useful beyond simple chat, they must hold sensitive user context such as calendars, credentials, health records, and financial data. We study whether the mere presence of such secrets in a model's context window introduces hidden correlations into the model's benign outputs, allowing reconstruction even when the model correctly refuses direct extraction. We further study whether an adversary can actively engineer prompts that amplify this effect, using the model as a covert carrier to transmit secrets through seemingly innocuous text. In both cases, this limited leakage is exploited using a novel adaptive attack that assumes black-box access to the underlying model. In controlled experiments across eight proprietary models, we find that 2-digit in-context secrets are reconstructed with near-perfect accuracy and 4-digit secrets at 82\% exact match, all from outputs the model produces in response to ordinary, non-adversarial requests. We observe that more capable models leak more: stronger instruction-following amplifies sensitivity to in-context secrets, suggesting leakage is a byproduct of capability as opposed to a patchable bug. We show this leakage enables two practical attacks: (1) a trained classifier that infers semantic predicates about user memories (e.g., health conditions, financial events) from routine natural-language outputs, and (2) an RL-trained adversary that extracts full Social Security Numbers from a production-style agent.

Problem

• Inadvertent Context Leakage
• Language Models
• Sensitive Information
• Black-box Access
• Adaptive Attack

Innovation

• inadvertent context leakage
• adaptive attack
• black-box access
• semantic predicates inference
• reinforcement learning