CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments

2026-04-07

AI Summary

This study addresses the limitations of existing large language model (LLM) cybersecurity evaluation frameworks, which predominantly target general IT environments and fail to accommodate the specialized protocols and operational constraints of industrial control systems such as IEC 61850-based digital substations. To bridge this gap, the authors propose CritBench—the first LLM security capability assessment framework tailored for IEC 61850 environments—encompassing 81 domain-specific tasks across scenarios including static configuration analysis, network traffic reconnaissance, and virtual machine interaction. The framework incorporates a dedicated tool scaffold to enable interaction with industrial protocols. Experimental results demonstrate that while LLMs perform well in static file parsing and single-step network enumeration, their capabilities are notably constrained in dynamic tasks requiring sustained reasoning and state tracking; however, integrating the tool scaffold substantially enhances their performance in complex operational tasks.

Abstract

The advancement of Large Language Models (LLMs) has raised concerns regarding their dual-use potential in cybersecurity. Existing evaluation frameworks overwhelmingly focus on Information Technology (IT) environments, failing to capture the constraints, and specialized protocols of Operational Technology (OT). To address this gap, we introduce CritBench, a novel framework designed to evaluate the cybersecurity capabilities of LLM agents within IEC 61850 Digital Substation environments. We assess five state-of-the-art models, including OpenAI's GPT-5 suite and open-weight models, across a corpus of 81 domain-specific tasks spanning static configuration analysis, network traffic reconnaissance, and live virtual machine interaction. To facilitate industrial protocol interaction, we develop a domain-specific tool scaffold. Our empirical results show that agents reliably execute static structured-file analysis and single-tool network enumeration, but their performance degrades on dynamic tasks. Despite demonstrating explicit, internalized knowledge of the IEC 61850 standards terminology, current models struggle with the persistent sequential reasoning and state tracking required to manipulate live systems without specialized tools. Equipping agents with our domain-specific tool scaffold significantly mitigates this operational bottleneck. Code and evaluation scripts are available at: https://github.com/GKeppler/CritBench

Problem

• Large Language Models
• Cybersecurity Evaluation
• IEC 61850
• Operational Technology
• Digital Substation

Innovation

• CritBench
• IEC 61850
• Operational Technology (OT)
• Large Language Models (LLMs)
• domain-specific tool scaffold