Learning Robust Penetration-Testing Policies under Partial Observability: A systematic evaluation

2025-09-24

AI Summary

This paper addresses the breakdown of Markovian assumptions and poor policy generalization/transferability in network penetration testing due to partial observability. To this end, we introduce the first randomized, partially observable penetration testing benchmark designed for realistic scenarios. Methodologically, we systematically compare historical state aggregation mechanisms—including frame stacking, RNNs, and Transformers—within Proximal Policy Optimization (PPO) and its variants, revealing the critical role of history modeling in policy learning; our optimal architecture accelerates convergence by up to 3×. Our contributions are threefold: (1) a more representative non-Markovian penetration testing environment; (2) empirical validation that historical modeling fundamentally enhances policy robustness and cross-scale transferability; and (3) behavior-level analysis yielding interpretable insights into agent decision-making beyond aggregate performance metrics.

Technology Category

• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Penetration testing, the simulation of cyberattacks to identify security vulnerabilities, presents a sequential decision-making problem well-suited for reinforcement learning (RL) automation. Like many applications of RL to real-world problems, partial observability presents a major challenge, as it invalidates the Markov property present in Markov Decision Processes (MDPs). Partially Observable MDPs require history aggregation or belief state estimation to learn successful policies. We investigate stochastic, partially observable penetration testing scenarios over host networks of varying size, aiming to better reflect real-world complexity through more challenging and representative benchmarks. This approach leads to the development of more robust and transferable policies, which are crucial for ensuring reliable performance across diverse and unpredictable real-world environments. Using vanilla Proximal Policy Optimization (PPO) as a baseline, we compare a selection of PPO variants designed to mitigate partial observability, including frame-stacking, augmenting observations with historical information, and employing recurrent or transformer-based architectures. We conduct a systematic empirical analysis of these algorithms across different host network sizes. We find that this task greatly benefits from history aggregation. Converging three times faster than other approaches. Manual inspection of the learned policies by the algorithms reveals clear distinctions and provides insights that go beyond quantitative results.

Problem

• Developing robust penetration testing policies under partial observability conditions
• Addressing sequential decision-making challenges in cybersecurity vulnerability identification
• Evaluating RL algorithms for handling network complexity and limited information

Innovation

• Uses PPO variants for partial observability
• Compares frame-stacking and recurrent architectures
• Employs history aggregation for faster convergence