SA-DRL: Security-Aware Deep Reinforcement Learning for Ransomware Detection with Asymmetric Reward Design

2026-07-07

AI Summary

This study addresses the limitations of traditional ransomware detection methods that employ symmetric loss functions, which fail to capture the asymmetric risks inherent in false negatives (high-cost) versus false positives (low-cost). To overcome this, the authors propose SA-DRL, a security-aware deep reinforcement learning framework that explicitly incorporates asymmetric security costs into the reward function. The approach introduces a Security-Optimal Model Selection (SOMS) criterion and an adaptive sample ranking strategy. Experimental results across DQN, DDQN, PPO, and A2C demonstrate that the optimal configuration (DDQN with reward function R2 and discount factor γ=0.1) achieves a false negative rate of 0.0080, an F1 score of 0.9915, and an AUC of 0.998—reducing false negatives by 67.6% compared to the best supervised baseline and lowering the average false negative rate by 43%.

Abstract

Ransomware detection is a security-critical task in which false negatives and false positives have unequal operational consequences. Conventional machine learning detectors often use symmetric objectives that penalize missed ransomware detections and benign false alarms equally, although a false negative can cause irreversible encryption, operational disruption, and high recovery cost, whereas a false positive is usually reversible. This study proposes a Security-Aware Deep Reinforcement Learning (SA-DRL) framework that embeds false-negative and false-positive cost asymmetry into the reinforcement learning reward signal to prioritize missed-detection reduction. The framework also introduces a Security-Optimal Model Selection (SOMS) criterion and an adaptive episode-level sample-ordering mechanism. Four deep reinforcement learning agents, DQN, DDQN, PPO, and A2C, were evaluated using a symmetric baseline reward (R1) and a security-aware asymmetric reward (R2). Experiments used four discount factors, five-fold cross-validation, and three random seeds, resulting in 480 training runs on a balanced ransomware detection dataset. The SOMS criterion selects models by prioritizing false-negative rate, followed by F1-score and training time. Results show that asymmetric reward shaping improves security-oriented detection performance. The SOMS-selected configuration, DDQN with R2 and gamma = 0.1, achieved a false-negative rate of 0.0080, an F1-score of 0.9915, and an AUC of 0.998, reducing missed detections by 67.6% compared with the best supervised baseline. Across all configurations, R2 reduced the mean false-negative rate by 43% relative to R1. These findings show that reward-function design is important for security-sensitive ransomware detection.

Problem

• ransomware detection
• false negative
• false positive
• asymmetric cost
• security-aware learning

Innovation

• asymmetric reward design
• security-aware reinforcement learning
• ransomware detection
• false-negative prioritization
• Security-Optimal Model Selection