TRAP: Benchmark for Task-completion and Resistance to Active Privacy-extraction

2026-06-17

AI Summary

This work addresses the inherent tension in document-centric tasks where agents must leverage private information to complete tasks while avoiding unintended output leakage. We introduce TRAP, the first benchmark to systematically evaluate large language models’ resilience against active privacy attacks when performing tasks that depend on private fields. Theoretical analysis demonstrates that soft constraints—such as prompting—cannot simultaneously achieve high task accuracy and zero leakage. To overcome this limitation, we propose a hash-based structural isolation mechanism that severs leakage pathways without compromising task performance. Evaluation across 22 prominent models reveals significant privacy vulnerabilities in all, with instruction-following capability positively correlated with leakage rates. Our method effectively mitigates these risks while preserving near-original task accuracy.

Abstract

Agents are increasingly deployed in document-intensive workflows where sensitive private information is not an edge case but a routine input, e.g., an agent booking a flight needs passport numbers. In such settings, the agent must use private information to complete tasks accurately while never exposing it in its responses, because it cannot verify who is actually at the keyboard. These two obligations are in fundamental tension. A model capable enough to use private information for task completion can, by the same capability, be induced to reveal it. To evaluate the trade-off of task accuracy and privacy leakage, we introduce Task-completion and Resistance to Active Privacy-extraction (TRAP). Each scenario includes a document containing private information, a task query that requires the agent to invoke the correct tool using private fields, and an attack query that attempts to elicit the same information in natural language. Evaluating 22 models spanning frontier proprietary and open-source models at multiple scales, we find that all model families exhibit non-trivial leakage, and that instruction-following ability correlates with leakage rate. Existing prompt-based defenses reduce leakage but at significant cost to task accuracy. Prompt optimization fails to escape this trade-off. We demonstrate that this failure is not incidental. For any softmax-based model, no soft-constraint defense, e.g., prompt-based defenses, can jointly achieve high task success with zero leakage probability. Motivated by this impossibility result, we propose structural private field isolation, which replaces private fields with hash keys before they reach the model. This approach largely prevents leakage while keeping task accuracy.

Problem

• privacy leakage
• task completion
• private information
• active privacy extraction
• agent safety

Innovation

• privacy leakage
• task completion
• active privacy extraction
• structural isolation
• impossibility result