Human-AI Collaboration in Cloud Security: Cognitive Hierarchy-Driven Deep Reinforcement Learning

2025-02-22

AI Summary

To address the challenges of real-time APT defense and weak human-AI collaboration in multi-tenant cloud environments, this paper proposes an adaptive Security Operations Center (SOC) decision-making framework grounded in Cognitive Hierarchy Theory (CHT). Methodologically, we first embed CHT into a Deep Q-Network (DQN), integrating attack graph modeling with Prospect Theory to dynamically characterize security analysts’ risk preferences and adversarial AI behaviors in a game-theoretic setting. We further design a human-in-the-loop (HITL) evaluation mechanism—MTurk HITL—to empirically validate interactive efficacy. Our theoretical contribution includes proving a tighter lower bound on Q-value convergence for the enhanced DQN. Simulation results demonstrate a 23.6% improvement in data protection rate and a 31.4% reduction in policy deviation. HITL experiments confirm significant gains in analyst decision consistency and risk responsiveness. Collectively, this work establishes a verifiable paradigm for trustworthy human-AI collaborative defense.

Technology Category

• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Multiagent Systems: Adversarial Agents

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Security and Privacy: Large-scale security measurements

Abstract

Given the complexity of multi-tenant cloud environments and the need for real-time threat mitigation, Security Operations Centers (SOCs) must integrate AI-driven adaptive defenses against Advanced Persistent Threats (APTs). However, SOC analysts struggle with countering adaptive adversarial tactics, necessitating intelligent decision-support frameworks. To enhance human-AI collaboration in SOCs, we propose a Cognitive Hierarchy Theory-driven Deep Q-Network (CHT-DQN) framework that models SOC analysts' decision-making against AI-driven APT bots. The SOC analyst (defender) operates at cognitive level-1, anticipating attacker strategies, while the APT bot (attacker) follows a level-0 exploitative policy. By incorporating CHT into DQN, our framework enhances SOC defense strategies via Attack Graph (AG)-based reinforcement learning. Simulation experiments across varying AG complexities show that CHT-DQN achieves higher data protection and lower action discrepancies compared to standard DQN. A theoretical lower bound analysis further validates its superior Q-value performance. A human-in-the-loop (HITL) evaluation on Amazon Mechanical Turk (MTurk) reveals that SOC analysts using CHT-DQN-driven transition probabilities align better with adaptive attackers, improving data protection. Additionally, human decision patterns exhibit risk aversion after failure and risk-seeking behavior after success, aligning with Prospect Theory. These findings underscore the potential of integrating cognitive modeling into deep reinforcement learning to enhance SOC operations and develop real-time adaptive cloud security mechanisms.

Problem

• Enhancing Human-AI collaboration in cloud security
• Mitigating Advanced Persistent Threats in real-time
• Integrating Cognitive Hierarchy Theory with reinforcement learning

Innovation

• Cognitive Hierarchy Theory-driven DQN
• Attack Graph-based reinforcement learning
• Human-in-the-loop evaluation