SubstratumGraphEnv: Reinforcement Learning Environment (RLE) for Modeling System Attack Paths

2026-03-01

AI Summary

This work proposes the first dynamic modeling framework that integrates graph structures with reinforcement learning to address the challenges of capturing sequentiality, interdependencies, and evolution in attack path identification within networked systems. The approach constructs a graph-based representation of operating system states from Sysmon logs and introduces a custom Gymnasium environment interfaced with PyTorch—termed SubstratumBridge—to seamlessly integrate Graph Convolutional Networks (GCNs) with the Advantage Actor-Critic (A2C) algorithm. This enables end-to-end automatic transformation of raw system logs into learnable representations of attack paths. The resulting graph-based reinforcement learning environment not only facilitates context-aware agent training but also uncovers the influence of critical system event attributes on attack path detection performance, thereby establishing a foundation for automated cybersecurity analysis.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Automating network security analysis, particularly the identification of potential attack paths, presents significant challenges. Due in part to the sequential, interconnected, and evolutionary nature of system events which most artificial intelligence (AI) techniques struggle to model effectively. This paper proposes a Reinforcement Learning (RL) environment generation framework that simulates the sequence of processes executed on a Windows operating system, enabling dynamic modeling of malicious processes on a system. This methodology models operating system state and transitions using a graph representation. This graph is derived from open-source System Monitor (Sysmon) logs. To address the variety in system event types, fields, and log formats, a mechanism was developed to capture and model parent-child processes from Sysmon logs. A Gymnasium environment (SubstratumGraphEnv) was constructed to establish the perceptible basis for an RL environment, and a customized PyTorch interface was also built (SubstratumBridge) to translate Gymnasium graphs into Deep Reinforcement Learning (DRL) observations and discrete actions. Graph Convolutional Networks (GCNs) concretize the graph's local and global state, which feed the distinct policy and critic heads of an Advantage Actor-Critic (A2C) model. This work's central contribution lies in the design of a novel deep graphical RL environment that automates translation of sequential user and system events, furnishing crucial context for cybersecurity analysis. This work provides a foundation for future research into shaping training parameters and advanced reward shaping, while also offering insight into which system events attributes are critical to training autonomous RL agents.

Problem

• attack path identification
• network security analysis
• system event modeling
• sequential system events
• cybersecurity automation

Innovation

• Graph Reinforcement Learning
• Attack Path Modeling
• Sysmon Log Processing
• Graph Convolutional Networks
• Gymnasium Environment