AutoDojo: Adaptive Attacks Expose Superficial Defenses and User-Underspecification Limits in LLM Agents

2026-06-12

AI Summary

Current defenses against indirect prompt injection (IPI) attacks on large language model (LLM) agents are predominantly evaluated under static benchmarks, which fail to capture their true robustness against adaptive adversaries. This work proposes AutoDojo, the first black-box evaluation framework that enables adaptive IPI attacks by iteratively generating adversarial prompts using state-of-the-art LLMs and uniformly assessing prompt-level, detection-level, and system-level defenses within the AgentDojo environment. Experimental results demonstrate that AutoDojo substantially increases attack success rates: even filters completely immune to static attacks exhibit an overall attack success rate (ASR) of 28%, rising to 64% in action-open tasks where user intent is not explicitly specified. These findings expose fundamental structural vulnerabilities in existing defense mechanisms.

Abstract

Indirect prompt injection (IPI) is a major security threat to LLM-powered agents. Thus, a growing body of work have proposed a variety of defensive approaches against IPI. These can be grouped into three broad categories: 1) prompt-based (using prompting as a way to prevent agents from following malicious instructions), 2) detection-based (identifying and filtering malicious instructions), and 3) system-level (using systems insights, such as control and data isolation, for defense). However, commonly used benchmarks for evaluating defense, such as AgentDojo, are \emph{inherently static}, generating a fixed distribution of IPI attacks. Consequently, static benchmarks do not usefully evaluate defense robustness to adaptive threats. We address this issue by developing AutoDojo, an adaptive extension of AgentDojo that optimizes IPI against a given defense. Using AutoDojo against state-of-the-art IPI defenses across three task suites and five target models, we make two key observations. First, many defenses offer only limited protection: a cheap, black-box adaptive attack using a frontier LLM to iteratively optimize the injection raises attack success rate (ASR) well above the level achieved by static injections against nearly all evaluated defenses. Against a filter that reduces static ASR to 0\%, AutoDojo recovers 28\% overall and 64\% on action-open tasks. Second, for prompt-level and filter-based defenses, ASR is substantially higher on \emph{action-open} tasks -- where the user's request delegates the action itself to attacker-controlled content -- than on precisely specified tasks. This is a structural limit: on such tasks the injection can pose as ordinary data rather than an explicit instruction, bypassing defenses that rely on detecting instruction-like text. AutoDojo is publicly available at https://github.com/xhOwenMa/AutoDojo.

Problem

• indirect prompt injection
• adaptive attacks
• LLM agents
• defense robustness
• user underspecification

Innovation

• adaptive attack
• indirect prompt injection
• LLM agent security
• action-open tasks
• defense robustness