A Framework for Evaluating Emerging Cyberattack Capabilities of AI

2025-03-14

AI Summary

Existing AI-driven cyberattack capability assessments lack systematicity, fail to cover the full attack lifecycle, and inadequately support precise defense. Method: This paper introduces the first end-to-end AI cyberattack capability assessment framework, innovatively adapting the cyber kill chain to AI systems. Grounded in over 12,000 real-world attack attempts, it distills seven AI-native attack patterns and constructs fifty cross-phase evaluation challenges. It proposes the novel “bottleneck analysis” method to identify AI-induced cost-disruption points and enables targeted defense prioritization and AI-augmented red teaming simulations. Results: Experiments reveal significant AI-amplification effects—particularly in initial access and privilege escalation phases. The work delivers (1) the most comprehensive AI cyber risk assessment framework to date, (2) an actionable defensive strategy guide, and (3) an open-source assessment toolkit.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Humans and AI: AI for Accessibility
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements

Abstract

As frontier models become more capable, the community has attempted to evaluate their ability to enable cyberattacks. Performing a comprehensive evaluation and prioritizing defenses are crucial tasks in preparing for AGI safely. However, current cyber evaluation efforts are ad-hoc, with no systematic reasoning about the various phases of attacks, and do not provide a steer on how to use targeted defenses. In this work, we propose a novel approach to AI cyber capability evaluation that (1) examines the end-to-end attack chain, (2) helps to identify gaps in the evaluation of AI threats, and (3) helps defenders prioritize targeted mitigations and conduct AI-enabled adversary emulation to support red teaming. To achieve these goals, we propose adapting existing cyberattack chain frameworks to AI systems. We analyze over 12,000 instances of real-world attempts to use AI in cyberattacks catalogued by Google's Threat Intelligence Group. Using this analysis, we curate a representative collection of seven cyberattack chain archetypes and conduct a bottleneck analysis to identify areas of potential AI-driven cost disruption. Our evaluation benchmark consists of 50 new challenges spanning different phases of cyberattacks. Based on this, we devise targeted cybersecurity model evaluations, report on the potential for AI to amplify offensive cyber capabilities across specific attack phases, and conclude with recommendations on prioritizing defenses. In all, we consider this to be the most comprehensive AI cyber risk evaluation framework published so far.

Problem

• Evaluates AI's emerging cyberattack capabilities systematically
• Identifies gaps in AI threat evaluation and defense prioritization
• Proposes a framework for AI-enabled adversary emulation and red teaming

Innovation

• Adapts cyberattack chain frameworks for AI systems
• Analyzes 12,000 AI cyberattack instances for insights
• Proposes 50 challenges for AI cyberattack evaluation