Distributed Detection of Adversarial Attacks in Multi-Agent Reinforcement Learning with Continuous Action Space

2025-08-21

AI Summary

This paper addresses the vulnerability of cooperative multi-agent reinforcement learning (MARL) systems in continuous action spaces to stealthy adversarial attacks. We propose a decentralized, local-observation-based distributed anomaly detection framework. Our method employs deep neural networks to parameterize multivariate Gaussian distributions modeling each agent’s normal behavior; continuous density estimation yields normality scores, which are monitored online via a bilateral CUSUM algorithm for real-time attack detection. Our key contribution lies in the tight integration of statistical process control with deep probabilistic modeling—enabling, for the first time, a scalable, fully decentralized detection mechanism for continuous-action MARL that requires no global information or centralized coordination. Extensive experiments on multiple PettingZoo benchmarks demonstrate robust performance: the proposed method achieves AUC-ROC scores exceeding 0.95 against strong interference attacks, significantly outperforming existing approaches designed for discrete-action MARL settings.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We address the problem of detecting adversarial attacks against cooperative multi-agent reinforcement learning with continuous action space. We propose a decentralized detector that relies solely on the local observations of the agents and makes use of a statistical characterization of the normal behavior of observable agents. The proposed detector utilizes deep neural networks to approximate the normal behavior of agents as parametric multivariate Gaussian distributions. Based on the predicted density functions, we define a normality score and provide a characterization of its mean and variance. This characterization allows us to employ a two-sided CUSUM procedure for detecting deviations of the normality score from its mean, serving as a detector of anomalous behavior in real-time. We evaluate our scheme on various multi-agent PettingZoo benchmarks against different state-of-the-art attack methods, and our results demonstrate the effectiveness of our method in detecting impactful adversarial attacks. Particularly, it outperforms the discrete counterpart by achieving AUC-ROC scores of over 0.95 against the most impactful attacks in all evaluated environments.

Problem

• Detecting adversarial attacks in multi-agent reinforcement learning
• Continuous action space challenges in cooperative agent systems
• Decentralized real-time anomaly detection using local observations

Innovation

• Decentralized detector using local observations
• Deep neural networks approximating Gaussian distributions
• Two-sided CUSUM procedure for real-time detection