Evaluating Jailbreaking Vulnerabilities in LLMs Deployed as Assistants for Smart Grid Operations: A Benchmark Against NERC Standards

2026-04-25

AI Summary

This study addresses the vulnerability of large language models (LLMs) in smart grid operations to prompt-based attacks that circumvent safety alignment mechanisms, potentially generating instructions violating North American Electric Reliability Corporation (NERC) reliability standards. For the first time, NERC standards are integrated with LLM jailbreaking techniques to establish an evaluation benchmark for critical infrastructure. The authors assess GPT-4o mini, Gemini 2.0 Flash-Lite, and Claude 3.5 Haiku across nine NERC standard scenarios—including EOP, TOP, and CIP—using three attack methods: Baseline, BitBypass, and DeepInception. Results reveal an overall attack success rate of 33.1%, with DeepInception achieving the highest rate at 63.17%. Claude demonstrated complete immunity, whereas Gemini proved most susceptible at 55.04%. Notably, fine-tuned prompts elevated simple attack success rates to 30.6%, highlighting significant inter-model security disparities.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Security and Privacy: Large-scale security measurements

Abstract

The deployment of Large Language Models (LLMs) as assistants in electric grid operations promises to streamline compliance and decision-making but exposes new vulnerabilities to prompt-based adversarial attacks. This paper evaluates the risk of jailbreaking LLMs, i.e., circumventing safety alignments to produce outputs violating regulatory standards, assuming threats from authorized users, such as operators, who craft malicious prompts to elicit non-compliant guidance. Three state-of-the-art LLMs (OpenAI's GPT-4o mini, Google's Gemini 2.0 Flash-Lite, and Anthropic's Claude 3.5 Haiku) were tested against Baseline, BitBypass, and DeepInception jailbreaking methods across scenarios derived from nine NERC Reliability Standards (EOP, TOP, and CIP). In the initial broad experiment, the overall Attack Success Rate (ASR) was 33.1%, with DeepInception proving most effective at 63.17% ASR. Claude 3.5 Haiku exhibited complete resistance (0% ASR), while Gemini 2.0 Flash-Lite was most vulnerable (55.04% ASR) and GPT-4o mini moderately susceptible (44.34% ASR). A follow-up experiment refining malicious wording in Baseline and BitBypass attacks yielded a 30.6% ASR, confirming that subtle prompt adjustments can enhance simpler methods' efficacy.

Problem

• jailbreaking
• Large Language Models
• smart grid
• NERC standards
• adversarial prompts

Innovation

• jailbreaking
• Large Language Models
• smart grid security
• NERC standards
• adversarial prompting