Mining Workflow Graphs for Black-Box Boundary Testing of Conversational LLM Agents

2026-07-07

AI Summary

Conversational large language model agents often incur security risks due to failures at state-dependent boundaries—such as authentication or confirmation checkpoints—that are concealed within multi-turn dialogues, which conventional testing methods struggle to cover. This work proposes AgentEval, a novel black-box testing framework that introduces dialogue workflow graph modeling for the first time. By interactively inferring workflow graphs from agent interactions, AgentEval identifies critical boundary conditions and replays dialogue paths with perturbations to generate targeted test cases. Requiring no access to source code, the approach substantially outperforms prompt-only baselines, achieving average coverage of 23–38 boundary conditions across four τ³-bench agents—compared to only 12 for baselines—while simultaneously reducing redundancy and false positive rates.

Abstract

Conversational LLM agents can cause real-world harm when their internal workflows fail, such as completing a transaction without confirmation. Testing these state-dependent failures is difficult because critical boundaries, such as identity checks and confirmation gates, are hidden behind multi-turn conversational prerequisites, rendering them inaccessible to standard tests. We present AgentEval, a black-box testing framework that discovers and stresses these stateful boundaries. AgentEval interacts with an agent to mine a \emph{conversational workflow graph}, a model of its behavior. Instead of prompting blindly, AgentEval uses this graph's structure to enumerate specific guards and prerequisites as test targets, replaying the conversational path to a boundary before applying a perturbation. AgentEval then executes each test, determining whether it passes or fails using only the conversation turns. We benchmark AgentEval against a privileged, white-box auditor with access to the agent's underlying source code, which AgentEval never sees. On four $τ^3$-bench agents, AgentEval successfully generates tests covering $23$--$38$ distinct boundaries per agent; ablation studies attribute the gain to the graph's structure: $23$ distinct boundaries versus $12$ with a prompt-only baseline, at lower duplicate and false-alarm rates.

Problem

• conversational LLM agents
• black-box testing
• workflow boundaries
• state-dependent failures
• boundary testing

Innovation

• conversational workflow graph
• black-box testing
• stateful boundary
• AgentEval
• LLM agent evaluation