Closed Loop Interactive Embodied Reasoning for Robot Manipulation

2024-04-23

arXiv.org

AI Summary

To address the challenge of enabling embodied agents to perform multi-step, multimodal complex tasks in dynamic physical environments, this paper introduces CLIER—the first closed-loop embodied reasoning system. CLIER employs a modular neuro-symbolic reasoning framework that jointly supports natural language instruction understanding, visual perception, non-visual property estimation (e.g., object weight), online belief updating, and action re-planning. We develop a MuJoCo-Blender co-simulation environment enabling high-fidelity physics interaction and photorealistic rendering. Furthermore, we establish the first benchmark encompassing ten categories of multi-step embodied reasoning tasks. Experiments demonstrate that CLIER achieves over 76% task success rate in simulation and 64% on a real robotic arm, significantly improving robustness against environmental disturbances, sensor noise, and actuation uncertainty, as well as cross-task generalization capability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Multiagent Systems: Multiagent Learning
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

Embodied reasoning systems integrate robotic hardware and cognitive processes to perform complex tasks typically in response to a natural language query about a specific physical environment. This usually involves changing the belief about the scene or physically interacting and changing the scene (e.g. 'Sort the objects from lightest to heaviest'). In order to facilitate the development of such systems we introduce a new simulating environment that makes use of MuJoCo physics engine and high-quality renderer Blender to provide realistic visual observations that are also accurate to the physical state of the scene. Together with the simulator we propose a new benchmark composed of 10 classes of multi-step reasoning scenarios that require simultaneous visual and physical measurements. Finally, we develop a new modular Closed Loop Interactive Reasoning (CLIER) approach that takes into account the measurements of non-visual object properties, changes in the scene caused by external disturbances as well as uncertain outcomes of robotic actions. We extensively evaluate our reasoning approach in simulation and in the real world manipulation tasks with a success rate above 76% and 64%, respectively.

Problem

• Integrates robotic hardware and cognitive processes for complex tasks.
• Develops a closed-loop system for interactive embodied reasoning.
• Addresses uncertain outcomes and environmental changes in robot manipulation.

Innovation

• Modular Closed Loop Interactive Embodied Reasoning approach
• Multi-modal reasoning and action planning
• Operates in closed loop responding to environmental changes