Evaluating Endpoint Detection Robustness Against Genetic Algorithm Driven Code Transformations

2026-07-08

AI Summary

Current endpoint detection systems exhibit insufficient robustness against adaptive code variants, making it difficult to accurately assess their adversarial resilience. This work proposes ShellForge, a novel framework that, for the first time, integrates multi-objective genetic algorithms with real-time feedback from antivirus (AV) and endpoint detection and response (EDR) systems to automatically generate functionally equivalent yet structurally diverse remote command execution payloads. By leveraging techniques such as syntactic transformation, encoding schemes, and structural rearrangement, ShellForge establishes a reproducible benchmark for evaluating the robustness of endpoint detection mechanisms. The framework exposes significant vulnerabilities in both signature-based and behavior-based detection approaches when confronted with adversarial variants, thereby providing an empirical foundation for improving defensive architectures.

Abstract

Post-compromise test variants are widely used in controlled security evaluation and endpoint robustness benchmarking. However, modern Antivirus (AV) and Endpoint Detection and Response (EDR) systems increasingly combine signature- and behavior-based detection, challenging the reliability of conventional detection pipelines under adaptive variation. This study introduces ShellForge, a Genetic Algorithm (GA)-driven framework that evolves post-compromise variants representative of remote command execution to generate functionally equivalent variants for systematic detection evaluation. ShellForge applies syntactic transformations, encoding schemes, and structural permutations guided by a multi-objective fitness function informed by AV and EDR detection feedback. We compare ShellForge against representative baseline transformation frameworks under identical sandbox configurations. Our findings highlight measurable robustness gaps in baseline signature- and behavior-oriented detection pipelines under controlled variant generation. In addition, we propose a reproducible benchmark for endpoint detection robustness evaluation, motivating the need for robustness-aware defensive monitoring and behavioral correlation.

Problem

• Endpoint Detection
• Robustness
• Genetic Algorithm
• Code Transformation
• Behavior-based Detection

Innovation

• Genetic Algorithm
• Endpoint Detection Robustness
• Code Transformation
• Behavior-based Detection
• ShellForge