SafePilot: A Framework for Assuring LLM-enabled Cyber-Physical Systems

2026-03-22

AI Summary

This work addresses the critical challenge of hallucination-induced safety violations in large language models (LLMs) deployed within cyber-physical systems, which can compromise system reliability. To mitigate this risk, the authors propose SafePilot, a novel end-to-end assurance framework that uniquely supports both attribute-based and temporal safety constraints. SafePilot employs a hierarchical neuro-symbolic architecture that dynamically guides LLMs toward generating verifiably safe plans through a pipeline comprising task complexity assessment, automatic translation of natural language instructions into formal specifications, hierarchical planning, and iterative refinement. Empirical evaluation across two representative scenarios demonstrates the framework’s effectiveness and adaptability, significantly enhancing the safety and regulatory compliance of LLM-driven cyber-physical systems.

Technology Category

• Natural Language Processing: Safety and Robustness
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large Language Models (LLMs), deep learning architectures with typically over 10 billion parameters, have recently begun to be integrated into various cyber-physical systems (CPS) such as robotics, industrial automation, and autopilot systems. The abstract knowledge and reasoning capabilities of LLMs are employed for tasks like planning and navigation. However, a significant challenge arises from the tendency of LLMs to produce "hallucinations" - outputs that are coherent yet factually incorrect or contextually unsuitable. This characteristic can lead to undesirable or unsafe actions in the CPS. Therefore, our research focuses on assuring the LLM-enabled CPS by enhancing their critical properties. We propose SafePilot, a novel hierarchical neuro-symbolic framework that provides end-to-end assurance for LLM-enabled CPS according to attribute-based and temporal specifications. Given a task and its specification, SafePilot first invokes a hierarchical planner with a discriminator that assesses task complexity. If the task is deemed manageable, it is passed directly to an LLM-based task planner with built-in verification. Otherwise, the hierarchical planner applies a divide-and-conquer strategy, decomposing the task into sub-tasks, each of which is individually planned and later merged into a final solution. The LLM-based task planner translates natural language constraints into formal specifications and verifies the LLM's output against them. If violations are detected, it identifies the flaw, adjusts the prompt accordingly, and re-invokes the LLM. This iterative process continues until a valid plan is produced or a predefined limit is reached. Our framework supports LLM-enabled CPS with both attribute-based and temporal constraints. Its effectiveness and adaptability are demonstrated through two illustrative case studies.

Problem

• Large Language Models
• Cyber-Physical Systems
• Hallucination
• Safety Assurance
• Formal Specification

Innovation

• neuro-symbolic framework
• LLM hallucination mitigation
• formal specification verification
• hierarchical task planning
• cyber-physical system assurance