ASLEval: Measuring Privacy Exposure Displacement in LLM Agent Sessions

2026-09-16

AI Summary

本文提出ASLEval框架，通过预注册隐藏目标集、测量所有可见出口并保留内部痕迹来解决多步骤会话中隐私暴露错位的问题。

Abstract

Privacy evaluations of tool-using LLM agents often inspect a designated action, final response, or attacker report. These local proxies can miss unauthorized exposure elsewhere in a multi-step session and lack common ground truth across outlets, reports, and tool paths. We introduce privacy exposure displacement, the mismatch between a local evaluation proxy and target-grounded session exposure, and ASLEval, an authorization-aware framework that pre-registers a hidden target set, measures all declared visible exits, and reserves internal traces for diagnosis. Across multiple enterprise-style environments and independently implemented runtimes, we observe three recurring patterns. An expected-outlet-only view misses 46.9% of exposure recovered by the visible-exit union; attacker self-reports combine omissions with high false discovery; and schema-aligned internal evidence usually precedes visible exposure at the request/probe level. Reducing model-visible returns changes this path but can eliminate normal-task success. Independent human review supports the adjudication pipeline while identifying harder console and candidate cases. These findings motivate benchmarks that declare the complete visible boundary, ground claims in pre-specified targets and authorization, and report privacy together with task utility.

Problem

• privacy evaluation
• LLM agent
• exposure displacement
• multi-step session

Innovation

• privacy exposure displacement
• ASLEval framework
• multi-step session