Where Do LLM-based Systems Break? A System-Level Security Framework for Risk Assessment and Treatment

2026-03-08

AI Summary

Existing security analysis methods struggle to systematically evaluate the compound risks facing safety-critical systems that integrate large language models (LLMs). This work proposes the first unified, system-level risk assessment framework that cohesively integrates traditional cybersecurity, adversarial machine learning, and prompt injection attacks. By combining system modeling, Attack-Defense Trees (ADTrees), and CVSS exploitability scoring, the framework enables structured analysis of multi-stage attack paths and identification of common bottleneck nodes. Validation in a healthcare scenario demonstrates that defenses targeting these critical nodes substantially reduce overall system exploitability. The proposed approach offers a generalizable risk governance process for safety-critical systems empowered by LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Language Models (LLMs) are increasingly integrated into safety-critical workflows, yet existing security analyses remain fragmented and often isolate model behavior from the broader system context. This work introduces a goal-driven risk assessment framework for LLM-powered systems that combines system modeling with Attack-Defense Trees (ADTrees) and Common Vulnerability Scoring System (CVSS)-based exploitability scoring to support structured, comparable analysis. We demonstrate the framework through a healthcare case study, modeling multi-step attack paths targeting intervention in medical procedures, leakage of electronic health record (EHR) data, and disruption of service availability. Our analysis indicates that threats spanning (i) conventional cyber, (ii) adversarial ML, and (iii) conversational attacks that manipulate prompts or context often consolidate into a small number of dominant paths and shared system choke points, enabling targeted defenses to yield meaningful reductions in path exploitability. By systematically comparing defense portfolios, we align these risks with established vulnerability management practices and provide a domain-agnostic workflow applicable to other LLM-enabled critical systems.

Problem

• Large Language Models
• system-level security
• risk assessment
• attack paths
• vulnerability management

Innovation

• LLM-based systems
• Attack-Defense Trees
• CVSS
• system-level security
• risk assessment