Assured Autonomy with Neuro-Symbolic Perception

2025-05-27

AI Summary

To address the poor interpretability and insufficient safety assurance of purely data-driven perception models in safety-critical cyber-physical systems, this paper proposes the Neural-Symbolic Perception Paradigm (NeuSPaPer). NeuSPaPer integrates knowledge distillation from vision foundation models with a lightweight scene graph generation (SGG) algorithm to construct a real-time neural-symbolic framework that online generates structured scene relationship graphs, enabling deep semantic understanding from raw pixels. It bridges the gap between low-level perception and high-level reasoning via end-to-end training jointly leveraging multimodal real-world data, physics-based simulation, and symbolic inference. Experiments demonstrate that NeuSPaPer significantly enhances robustness, interpretability, and safety assurance of autonomous systems under adversarial conditions. By unifying neural perception with symbolic reasoning in a verifiable, structured representation, it establishes a novel paradigm for trustworthy autonomy.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Neuro-Symbolic Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Federated Web and WoT systems, including distributed, federated and edge-based data processing

Abstract

Many state-of-the-art AI models deployed in cyber-physical systems (CPS), while highly accurate, are simply pattern-matchers.~With limited security guarantees, there are concerns for their reliability in safety-critical and contested domains. To advance assured AI, we advocate for a paradigm shift that imbues data-driven perception models with symbolic structure, inspired by a human's ability to reason over low-level features and high-level context. We propose a neuro-symbolic paradigm for perception (NeuSPaPer) and illustrate how joint object detection and scene graph generation (SGG) yields deep scene understanding.~Powered by foundation models for offline knowledge extraction and specialized SGG algorithms for real-time deployment, we design a framework leveraging structured relational graphs that ensures the integrity of situational awareness in autonomy. Using physics-based simulators and real-world datasets, we demonstrate how SGG bridges the gap between low-level sensor perception and high-level reasoning, establishing a foundation for resilient, context-aware AI and advancing trusted autonomy in CPS.

Problem

• Enhancing AI reliability in safety-critical cyber-physical systems
• Bridging low-level perception with high-level symbolic reasoning
• Ensuring integrity of situational awareness in autonomous systems

Innovation

• Neuro-symbolic paradigm for perception (NeuSPaPer)
• Joint object detection and scene graph generation
• Structured relational graphs for situational awareness