LLMs in the SOC: An Empirical Study of Human-AI Collaboration in Security Operations Centres

2025-08-26

AI Summary

This study addresses the lack of clarity regarding human–LLM collaboration mechanisms in Security Operations Centers (SOCs). Drawing on a 10-month longitudinal dataset of 3,090 real-world analyst queries from 45 security professionals, it employs a mixed-methods approach—integrating the NICE Cybersecurity Workforce Framework, large-scale log mining, and qualitative–quantitative analysis—to systematically characterize LLM query intent, interaction patterns, and usage evolution. Results reveal an emergent, on-demand cognitive assistance paradigm: 93% of queries map to standardized cybersecurity competencies; most involve low-level telemetry interpretation and concise communication refinement. This indicates a shift from experimental tool to routine collaborative partner for a subset of analysts—particularly in threat sensing and technical articulation. The findings provide empirically grounded theoretical insights and actionable design principles for operationally viable AI-augmented SOCs.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The integration of Large Language Models (LLMs) into Security Operations Centres (SOCs) presents a transformative, yet still evolving, opportunity to reduce analyst workload through human-AI collaboration. However, their real-world application in SOCs remains underexplored. To address this gap, we present a longitudinal study of 3,090 analyst queries from 45 SOC analysts over 10 months. Our analysis reveals that analysts use LLMs as on-demand aids for sensemaking and context-building, rather than for making high-stakes determinations, preserving analyst decision authority. The majority of queries are related to interpreting low-level telemetry (e.g., commands) and refining technical communication through short (1-3 turn) interactions. Notably, 93% of queries align with established cybersecurity competencies (NICE Framework), underscoring the relevance of LLM use for SOC-related tasks. Despite variations in tasks and engagement, usage trends indicate a shift from occasional exploration to routine integration, with growing adoption and sustained use among a subset of analysts. We find that LLMs function as flexible, on-demand cognitive aids that augment, rather than replace, SOC expertise. Our study provides actionable guidance for designing context-aware, human-centred AI assistance in security operations, highlighting the need for further in-the-wild research on real-world analyst-LLM collaboration, challenges, and impacts.

Problem

• Empirical study of LLM integration in SOCs
• Analyzes human-AI collaboration for security tasks
• Investigates real-world analyst usage patterns and impacts

Innovation

• Human-AI collaboration for security operations
• LLMs as on-demand cognitive aids for analysts
• Context-aware AI assistance design for SOCs