LLM-enabled Applications Require System-Level Threat Monitoring

2026-02-23

AI Summary

This study addresses the expanded attack surface of large language model (LLM) applications, which stems from their non-deterministic behavior and inherent difficulty in verification, compounded by the absence of effective system-level threat monitoring mechanisms. The work introduces a system-level threat-aware framework that treats security threats as an intrinsic aspect of LLM application operation. By integrating runtime anomaly detection, context-aware analysis, and security event correlation, the framework enables continuous, contextualized monitoring of security-relevant anomalies. Moving beyond conventional static defense paradigms reliant on testing or guardrails, this approach establishes runtime monitoring as a foundational component of secure LLM deployment and provides theoretical grounding and strategic direction for developing specialized incident response systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Security and Privacy: Security and privacy of machine learning and AI applications
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

LLM-enabled applications are rapidly reshaping the software ecosystem by using large language models as core reasoning components for complex task execution. This paradigm shift, however, introduces fundamentally new reliability challenges and significantly expands the security attack surface, due to the non-deterministic, learning-driven, and difficult-to-verify nature of LLM behavior. In light of these emerging and unavoidable safety challenges, we argue that such risks should be treated as expected operational conditions rather than exceptional events, necessitating a dedicated incident-response perspective. Consequently, the primary barrier to trustworthy deployment is not further improving model capability but establishing system-level threat monitoring mechanisms that can detect and contextualize security-relevant anomalies after deployment -- an aspect largely underexplored beyond testing or guardrail-based defenses. Accordingly, this position paper advocates systematic and comprehensive monitoring of security threats in LLM-enabled applications as a prerequisite for reliable operation and a foundation for dedicated incident-response frameworks.

Problem

• LLM-enabled applications
• security attack surface
• system-level threat monitoring
• reliability challenges
• incident-response

Innovation

• system-level monitoring
• LLM security
• incident response
• threat detection
• reliable deployment