Hallucination-Resistant Security Planning with a Large Language Model

2026-02-05

AI Summary

This work addresses the unreliability of large language models (LLMs) in safety-critical planning due to hallucination-induced decision errors. To mitigate this issue, the authors propose an iterative refinement framework that validates LLM-generated candidate actions against system constraints and forward-looking predictions via an adjustable consistency threshold. When consistency falls below the threshold, external feedback—such as from a digital twin—is incorporated, and in-context learning is leveraged to refine action generation. Theoretical analysis establishes an upper regret bound for the in-context learning component, while empirical evaluation across four public datasets demonstrates the approach’s efficacy: compared to state-of-the-art LLMs, the proposed method reduces system recovery time by up to 30%.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Multiagent Systems: Multiagent Planning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Large language models (LLMs) are promising tools for supporting security management tasks, such as incident response planning. However, their unreliability and tendency to hallucinate remain significant challenges. In this paper, we address these challenges by introducing a principled framework for using an LLM as decision support in security management. Our framework integrates the LLM in an iterative loop where it generates candidate actions that are checked for consistency with system constraints and lookahead predictions. When consistency is low, we abstain from the generated actions and instead collect external feedback, e.g., by evaluating actions in a digital twin. This feedback is then used to refine the candidate actions through in-context learning (ICL). We prove that this design allows to control the hallucination risk by tuning the consistency threshold. Moreover, we establish a bound on the regret of ICL under certain assumptions. To evaluate our framework, we apply it to an incident response use case where the goal is to generate a response and recovery plan based on system logs. Experiments on four public datasets show that our framework reduces recovery times by up to 30% compared to frontier LLMs.

Problem

• hallucination
• security planning
• large language models
• reliability
• incident response

Innovation

• hallucination-resistant
• iterative decision framework
• in-context learning
• digital twin feedback
• consistency checking