VisualPredicator: Learning Abstract World Models with Neuro-Symbolic Predicates for Robot Planning

2024-10-30

arXiv.org

AI Summary

This paper addresses the challenge in robotic task planning where world models struggle to simultaneously achieve interpretability, generalization, and task-oriented abstraction. We propose a neuro-symbolic predicate-driven abstract world model. Methodologically, we introduce the first neuro-symbolic predicate language supporting *online predicate invention*, and design an end-to-end differentiable neuro-symbolic integration architecture that enables semantic compression of perception-action spaces and task-driven abstraction, jointly learning the abstract model and predicate representations. Our contributions are threefold: (1) the first integration of online predicate invention into world model learning; (2) a unified framework reconciling symbolic interpretability with neural representation generalization; and (3) significant improvements in sample efficiency, out-of-distribution generalization, and decision interpretability across five simulated robotic tasks—outperforming hierarchical reinforcement learning, vision-language model–based planning, and purely symbolic approaches.

Technology Category

• Machine Learning: Neuro-Symbolic Learning
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Broadly intelligent agents should form task-specific abstractions that selectively expose the essential elements of a task, while abstracting away the complexity of the raw sensorimotor space. In this work, we present Neuro-Symbolic Predicates, a first-order abstraction language that combines the strengths of symbolic and neural knowledge representations. We outline an online algorithm for inventing such predicates and learning abstract world models. We compare our approach to hierarchical reinforcement learning, vision-language model planning, and symbolic predicate invention approaches, on both in- and out-of-distribution tasks across five simulated robotic domains. Results show that our approach offers better sample complexity, stronger out-of-distribution generalization, and improved interpretability.

Problem

• Develops Neuro-Symbolic Predicates for task-specific abstractions.
• Compares with existing methods in robotic planning tasks.
• Improves sample complexity and out-of-distribution generalization.

Innovation

• Neuro-Symbolic Predicates combine symbolic and neural representations
• Online algorithm for inventing predicates and learning world models
• Improved sample complexity, generalization, and interpretability demonstrated