ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents

2026-08-10

AI Summary

This work addresses behavioral safety risks in collaborative agents, which may inadvertently leak sensitive data, manipulate internal states, or misuse APIs during task execution. To this end, the paper introduces the concept of “behavioral safety” and presents ActBench—the first self-evolving benchmark grounded in execution trajectories—that evaluates agents by injecting reachable payloads via adversarial variants alongside original tasks under identical configurations. Methodologically, it proposes a novel reward-guided beam search to jointly optimize attack effectiveness and task utility, complemented by an execution checkpoint reflection mechanism and dual-evidence verification leveraging both system logs and LLM-generated trajectories to ensure evaluation reliability. Experiments span 15 risk behaviors, 6 execution spaces, and 48 Web APIs, evaluating 15 large language models and 6 open-source agents across 24,000 trajectories; attack success rates range from 10.1% to 94.4%, revealing that model choice exerts a significantly greater impact on safety than agent framework design.

Abstract

Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API. We define behavioral safety and introduce ActBench, a self-evolving benchmark that evaluates such behavior risk from execution trajectories rather than final responses. Each case pairs a benign task with an adversarial variant that preserves its instruction, configuration, initial state, rating model, and trusted records while injecting a task-reachable payload. ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs.To move beyond static payloads, we propose a reward-guided beam search method that jointly optimizes attack effectiveness and task utility, while reflection diagnoses failed execution checkpoint and guides payload revision. Besides, we propose a dual evidence verification mechanism that verifies agent execution safety and utility through log evidence and LLM-based trajectory evidence.We evaluate 15 LLMs and 6 open-source cowork agents over 24,000 trajectories. Under a fixed harness, attack success rates ranges from 10.1% to 94.4% across models, while under a fixed base model, they range from 73.7% to 94.4% across agents.These results show greater variation across models than agent harness, while attacks remain highly successful across all tested harnesses.Our benchmark is released at: https://github.com/zjuicsr/ActBench.

Problem

• behavioral safety
• cowork agents
• execution trajectories
• adversarial payloads
• risk behaviors

Innovation

• behavioral safety
• self-evolving benchmark
• execution trajectory
• reward-guided beam search
• dual evidence verification