Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

2026-05-21

AI Summary

This study addresses the limitations of existing language model safety evaluations, which predominantly focus on static outputs and fail to capture progressive security risks emerging from state evolution during multi-turn interactions. To bridge this gap, the authors propose the first evaluation framework inspired by the “boiling frog” metaphor, specifically designed for office and enterprise settings. The framework introduces a multi-turn agent safety benchmark featuring a three-tier operational risk taxonomy, aligned with high-risk scenarios in the EU AI Act and general-purpose AI behavior guidelines. By incorporating stateful interaction chains, controlled risk payload injection, and a workspace-state–based safety scoring mechanism, the evaluation of nine mainstream models reveals an average strict attack success rate of 44.4%, with Gemini 3.1 Flash Lite reaching 92.9%. Notably, in loss-of-control scenarios, the average success rate climbs to 93.3%, underscoring the widespread vulnerability of current tool-augmented agents to progressive adversarial attacks.

Abstract

Background. Traditional safety benchmarks for language models evaluate generated text: whether a model outputs toxic language, reproduces bias, or follows harmful instructions. When models are deployed as agents, the safety-relevant object shifts from what the system says to what it does within an environment, and evaluating model responses under prompting is no longer sufficient to address the safety challenges posed by artificial intelligence. Recent developments have seen the rise of benchmarks that evaluate large language models as agents. We contribute to this strand of research. Approach. We introduce Boiling the Frog, a benchmark that evaluates whether tool-using AI models deployed in corporate and office settings are susceptible to incremental attacks. Each scenario begins with benign workspace edits and later introduces a risk-bearing request. The benchmark focuses on stateful multi-turn evaluation: chains expose a persistent workspace, place the risk-bearing payload at controlled positions in the turn sequence, and score whether the resulting artifact state becomes unsafe. Scenarios are organized through a three-level operational risk taxonomy grounded in the Boiling the Frog risks, the AI Act Annex I and Annex III high-risk contexts, and EU AI Act's Code of Practice on General-Purpose AI (GPAI). Results. Across a nine-model panel, aggregate strict attack success rate (ASR) is 44.4%. Model-level ASR ranges from 20.5% for Claude Haiku 4.5 to 92.9% for Gemini 3.1 Flash Lite, with Seed 2.0 Lite also above 80%. Average chain category-level ASR reaches 93.3% for Code of Practice loss-of-control scenarios.

Problem

• agentic safety
• incremental attacks
• multi-turn evaluation
• tool-using AI
• operational risk

Innovation

• agentic safety
• multi-turn benchmark
• incremental attacks
• stateful evaluation
• tool-using AI