Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

2026-07-16

AI Summary

This study addresses a critical gap in the evaluation of security agents, which has traditionally emphasized success rates while overlooking the operational costs associated with reasoning steps, tool invocations, and telemetry queries—factors essential to real-world economic efficiency. To bridge this gap, the authors propose the first cost-aware evaluation framework tailored for security operations, integrating Cybench offensive tasks and Splunk BOTS v1 defensive scenarios under fixed resource budgets. The framework enables fine-grained quantification of computational and query-related overheads. Findings reveal that open-source large language models can match or surpass proprietary systems in offensive tasks at lower cost, whereas defensive performance hinges critically on efficient tool utilization rather than raw computational power. These insights, disseminated via an interactive website, uncover fundamental differences in scaling dynamics between red- and blue-team operations.

Abstract

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.

Problem

• security agents
• cost-aware evaluation
• offensive security
• defensive security
• operational efficiency

Innovation

• cost-aware evaluation
• security agents
• language models
• offensive and defensive tasks
• operational efficiency