Less is more? Rewards in RL for Cyber Defence

2025-03-05

AI Summary

Dense reward signals in training autonomous network defense agents often lead to suboptimal defensive policies. Method: We propose and empirically validate a sparse reward mechanism, featuring (1) a benchmark evaluation score grounded in real-world security states; (2) two sparse reward formulations—emphasizing positive sparse rewards (e.g., “network uncompromised”); and (3) experiments within a multi-scale Cyber Gym–adapted network simulation environment (2–50 nodes), integrating deep reinforcement learning with explicit modeling of proactive and reactive defense actions. Results: Sparse rewards significantly improve agent defense effectiveness, robustness, and training stability. These gains remain consistent across diverse network topologies and attack-defense scenarios, consistently outperforming state-of-the-art dense reward baselines. This work constitutes the first systematic demonstration of the superiority of sparse reward design for autonomous network defense, establishing a novel paradigm for reward engineering in cyber-physical security domains.

Technology Category

• Multiagent Systems: Adversarial Agents
• Machine Learning: Online Learning & Bandits
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The last few years has seen an explosion of interest in autonomous cyber defence agents based on deep reinforcement learning. Such agents are typically trained in a cyber gym environment, also known as a cyber simulator, at least 32 of which have already been built. Most, if not all cyber gyms provide dense"scaffolded"reward functions which combine many penalties or incentives for a range of (un)desirable states and costly actions. Whilst dense rewards help alleviate the challenge of exploring complex environments, yielding seemingly effective strategies from relatively few environment steps; they are also known to bias the solutions an agent can find, potentially towards suboptimal solutions. Sparse rewards could offer preferable or more effective solutions and have been overlooked by cyber gyms to date. In this work we set out to evaluate whether sparse reward functions might enable training more effective cyber defence agents. Towards this goal we first break down several evaluation limitations in existing work by proposing a ground truth evaluation score that goes beyond the standard RL paradigm used to train and evaluate agents. By adapting a well-established cyber gym to accommodate our methodology and ground truth score, we propose and evaluate two sparse reward mechanisms and compare them with a typical dense reward. Our evaluation considers a range of network sizes, from 2 to 50 nodes, and both reactive and proactive defensive actions. Our results show that sparse rewards, particularly positive reinforcement for an uncompromised network state, enable the training of more effective cyber defence agents. Furthermore, we show that sparse rewards provide more stable training than dense rewards, and that both effectiveness and training stability are robust to a variety of cyber environment considerations.

Problem

• Evaluates sparse vs dense rewards in RL for cyber defense.
• Proposes ground truth score for better agent evaluation.
• Shows sparse rewards improve training stability and effectiveness.

Innovation

• Sparse rewards improve cyber defence agent training.
• Ground truth score enhances RL evaluation accuracy.
• Positive reinforcement for uncompromised networks boosts effectiveness.