SentinelAgent: Graph-based Anomaly Detection in Multi-Agent Systems

2025-05-30

AI Summary

LLM-driven multi-agent systems (MAS) face systemic security risks—including prompt injection, tool misuse, and erroneous coordination—where existing defenses, confined to I/O-level safeguards, fail to address distributed, dynamic failures. This paper proposes an anomaly detection framework integrating structural modeling and runtime monitoring. It introduces a novel dynamic directed execution graph (DDEG) for multi-granular semantic anomaly detection at node, edge, and path levels. We further design SentinelAgent, a pluggable LLM-based supervisory agent that enables policy-driven intervention and root-cause explainable attribution. The method synergizes graph neural networks, LLM-based safety policy reasoning, and multi-agent trajectory analysis. Evaluated on an email assistant and the Magentic-One system, our approach successfully uncovers covert collaborative attacks and latent exploitation paths, achieving significant improvements in detection accuracy and attribution interpretability.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Agentic search

Abstract

The rise of large language model (LLM)-based multi-agent systems (MAS) introduces new security and reliability challenges. While these systems show great promise in decomposing and coordinating complex tasks, they also face multi-faceted risks across prompt manipulation, unsafe tool usage, and emergent agent miscoordination. Existing guardrail mechanisms offer only partial protection, primarily at the input-output level, and fall short in addressing systemic or multi-point failures in MAS. In this work, we present a system-level anomaly detection framework tailored for MAS, integrating structural modeling with runtime behavioral oversight. Our approach consists of two components. First, we propose a graph-based framework that models agent interactions as dynamic execution graphs, enabling semantic anomaly detection at node, edge, and path levels. Second, we introduce a pluggable SentinelAgent, an LLM-powered oversight agent that observes, analyzes, and intervenes in MAS execution based on security policies and contextual reasoning. By bridging abstract detection logic with actionable enforcement, our method detects not only single-point faults and prompt injections but also multi-agent collusion and latent exploit paths. We validate our framework through two case studies, including an email assistant and Microsoft's Magentic-One system, demonstrating its ability to detect covert risks and provide explainable root-cause attribution. Our work lays the foundation for more trustworthy, monitorable, and secure agent-based AI ecosystems.

Problem

• Detects anomalies in multi-agent system interactions
• Addresses systemic and multi-point security failures
• Identifies agent collusion and latent exploit paths

Innovation

• Graph-based framework models agent interactions dynamically
• LLM-powered SentinelAgent observes and intervenes in execution
• Detects multi-agent collusion and latent exploit paths