Adaptive alert prioritisation in security operations centres via learning to defer with human feedback

2025-06-23

AI Summary

Security Operations Centers (SOCs) suffer from alert overload and inefficient prioritization; conventional predictive AI models fail to detect novel threats, while static human-AI collaboration frameworks (e.g., Learning-to-Defer, L2D) lack mechanisms for continuous adaptation via human feedback. This paper proposes Learning-to-Defer with Human Feedback (L2DHF), the first framework integrating deep reinforcement learning with human feedback (DRLHF) to enable an evolvable, dynamic delegation mechanism. L2DHF initializes its policy via supervised learning and refines the AI’s decision boundary in real time through online learning, achieving adaptive alert classification and intelligent triage. Evaluated on UNSW-NB15 and CICIDS2017, L2DHF improves critical-alert ranking accuracy by 13–67%, reduces high-risk misprioritization by 98%, and decreases manual delegation volume by 37%. These gains significantly enhance operational responsiveness and practical deployability in real-world SOCs.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Search and Optimization: Learning to Search
• Machine Learning: Hardware-aware ML

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Alert prioritisation (AP) is crucial for security operations centres (SOCs) to manage the overwhelming volume of alerts and ensure timely detection and response to genuine threats, while minimising alert fatigue. Although predictive AI can process large alert volumes and identify known patterns, it struggles with novel and evolving scenarios that demand contextual understanding and nuanced judgement. A promising solution is Human-AI teaming (HAT), which combines human expertise with AI's computational capabilities. Learning to Defer (L2D) operationalises HAT by enabling AI to "defer" uncertain or unfamiliar cases to human experts. However, traditional L2D models rely on static deferral policies that do not evolve with experience, limiting their ability to learn from human feedback and adapt over time. To overcome this, we introduce Learning to Defer with Human Feedback (L2DHF), an adaptive deferral framework that leverages Deep Reinforcement Learning from Human Feedback (DRLHF) to optimise deferral decisions. By dynamically incorporating human feedback, L2DHF continuously improves AP accuracy and reduces unnecessary deferrals, enhancing SOC effectiveness and easing analyst workload. Experiments on two widely used benchmark datasets, UNSW-NB15 and CICIDS2017, demonstrate that L2DHF significantly outperforms baseline models. Notably, it achieves 13-16% higher AP accuracy for critical alerts on UNSW-NB15 and 60-67% on CICIDS2017. It also reduces misprioritisations, for example, by 98% for high-category alerts on CICIDS2017. Moreover, L2DHF decreases deferrals, for example, by 37% on UNSW-NB15, directly reducing analyst workload. These gains are achieved with efficient execution, underscoring L2DHF's practicality for real-world SOC deployment.

Problem

• Adaptive alert prioritisation in security operations centres
• Learning to defer with human feedback for better AI decisions
• Reducing alert fatigue and improving threat detection accuracy

Innovation

• Learning to Defer with Human Feedback (L2DHF)
• Deep Reinforcement Learning from Human Feedback (DRLHF)
• Dynamic deferral policies via human feedback