Measuring AI Agents' Progress on Multi-Step Cyber Attack Scenarios

2026-03-11

AI Summary

This study systematically evaluates the autonomous offensive capabilities of state-of-the-art AI models in complex, multi-stage cyberattacks involving heterogeneous action sequences. To this end, two specialized network ranges—a 32-step enterprise network and a 7-step industrial control system attack scenario—were constructed, and seven large language models released over an 18-month period were tested under varying inference compute budgets. The work introduces the first automated evaluation framework based on multi-step attack ranges, integrating heterogeneous attack action modeling with high-compute inference scheduling to quantitatively disentangle the independent effects of model generation advances and inference compute on long-chain attack performance. Results show that model performance scales log-linearly with inference compute: the latest models achieve an average of 9.8 out of 32 steps (peaking at 22) under a 10M-token budget and demonstrate, for the first time, partially stable execution in industrial control scenarios (averaging 1.2–1.4 out of 7 steps).

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We evaluate the autonomous cyber-attack capabilities of frontier AI models on two purpose-built cyber ranges-a 32-step corporate network attack and a 7-step industrial control system attack-that require chaining heterogeneous capabilities across extended action sequences. By comparing seven models released over an eighteen-month period (August 2024 to February 2026) at varying inference-time compute budgets, we observe two capability trends. First, model performance scales log-linearly with inference-time compute, with no observed plateau-increasing from 10M to 100M tokens yields gains of up to 59%, requiring no specific technical sophistication from the operator. Second, each successive model generation outperforms its predecessor at fixed token budgets: on the corporate network range, average steps completed at 10M tokens rose from 1.7 (GPT-4o, August 2024) to 9.8 (Opus 4.6, February 2026). The best single run completed 22 of 32 steps, corresponding to roughly 6 of the estimated 14 hours a human expert would need. On the industrial control system range, performance remains limited, though the most recent models are the first to reliably complete steps, averaging 1.2-1.4 of 7 (max 3).

Problem

• AI agents
• cyber attack
• multi-step scenarios
• autonomous capabilities
• capability evaluation

Innovation

• multi-step cyber attack
• autonomous AI agents
• inference-time compute scaling
• cyber range evaluation
• capability progression