AgentLSD: Evaluating AI Security Agents Under Adversarial Task Contamination

2026-09-16

AI Summary

研究通过AgentLSD框架评估AI安全代理在对抗性任务污染下的表现，使用CTF挑战作为实验环境，揭示了欺骗性证据对AI性能的影响。

Abstract

AI agents for security inspect web pages, source code, logs, configuration files, and command outputs. These environments may contain deceptive artifacts that influence the agent's behavior. We call this adversarial task contamination. Whereas prompt injection relies on attacker-supplied instructions, task contamination also includes non-instructional evidence, such as fake results and decoy endpoints. We present AgentLSD, a controlled framework for studying adversarial task contamination. AgentLSD uses Capture the Flag (CTF) challenges as its experimental environment. We inject trap artifacts, such as fake flags, misleading hints, decoy endpoints, and hidden cues, while preserving the intended CTF solution. The framework supports paired clean and trap-augmented experiments with deterministic trap generation, runtime injection, telemetry, and delivery verification. We evaluate six models on 11 web CTF challenges. In the clean condition, agents capture 41% of the flags, and no model solves every challenge. We then measure the impact of task contamination. Even when the agent still recovers the flag, traps increase the number of turns (+20) and reasoning tokens (+2k). Solve-rate effects are more heterogeneous, as some model-challenge pairs are largely unaffected while others follow decoys or submit wrong flags. These results show that clean CTF performance understates vulnerability to deceptive task evidence. AgentLSD isolates this effect and provides a reproducible benchmark for studying it. We release the framework, configurations, trap specifications, and raw traces.

Problem

• adversarial task contamination
• AI security agents
• deceptive artifacts

Innovation

• Adversarial Task Contamination
• AgentLSD Framework
• Capture the Flag Challenges
• Trap Artifacts Injection
• Deterministic Trap Generation