MuBlE: MuJoCo and Blender simulation Environment and Benchmark for Task Planning in Robot Manipulation

2025-03-04

AI Summary

Existing embodied reasoning agents exhibit limited performance on long-horizon manipulation tasks requiring physical interaction for information acquisition (e.g., “sort objects by weight”) and suffer from poor generalization beyond their training environments. Method: We introduce the first high-fidelity simulation platform for long-horizon robotic manipulation, integrating MuJoCo’s accurate physics modeling with Blender’s photorealistic rendering to enable dual closed-loop interaction—vision-action and control-physics. Contribution/Results: Our platform is the first to jointly achieve high physical fidelity and visual realism for long-sequence tasks. We propose SHOP-VRB2, a benchmark comprising 10 multi-step reasoning scenarios that demand coordinated visual and physical measurements. Built upon robosuite, the platform is open-sourced to support multimodal data generation and closed-loop policy training. Experiments demonstrate substantial improvements in agents’ planning and reasoning capabilities on complex manipulation tasks, establishing critical infrastructure for long-horizon embodied reasoning systems.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Visual Reasoning & Symbolic Representations
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Economics and fairness of platforms and recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Current embodied reasoning agents struggle to plan for long-horizon tasks that require to physically interact with the world to obtain the necessary information (e.g. 'sort the objects from lightest to heaviest'). The improvement of the capabilities of such an agent is highly dependent on the availability of relevant training environments. In order to facilitate the development of such systems, we introduce a novel simulation environment (built on top of robosuite) that makes use of the MuJoCo physics engine and high-quality renderer Blender to provide realistic visual observations that are also accurate to the physical state of the scene. It is the first simulator focusing on long-horizon robot manipulation tasks preserving accurate physics modeling. MuBlE can generate mutlimodal data for training and enable design of closed-loop methods through environment interaction on two levels: visual - action loop, and control - physics loop. Together with the simulator, we propose SHOP-VRB2, a new benchmark composed of 10 classes of multi-step reasoning scenarios that require simultaneous visual and physical measurements.

Problem

• Develops simulation for long-horizon robot manipulation tasks
• Integrates MuJoCo physics and Blender for realistic visuals
• Introduces SHOP-VRB2 benchmark for multi-step reasoning scenarios

Innovation

• MuJoCo and Blender for realistic simulations
• Multimodal data generation for training
• Closed-loop methods via visual-action and control-physics loops