ManiSkill-HAB: A Benchmark for Low-Level Manipulation in Home Rearrangement Tasks

2024-12-09

arXiv.org

AI Summary

To address the urgent need for an efficient low-level manipulation benchmark for embodied AI in household object rearrangement—requiring high-speed simulation, complex environment modeling, and large-scale safe demonstration data—this paper introduces HomeAssistant-Bench, the first GPU-accelerated low-level rearrangement benchmark tailored to home environments. Methodologically: (1) we leverage a GPU-accelerated physics engine to achieve over 3× simulation speedup and significant memory optimization; (2) we propose a rule-driven trajectory filtering mechanism to ensure safety and controllability of large-scale demonstrations; and (3) we develop a unified data generation framework integrating both reinforcement learning and imitation learning baselines. Our contribution is the first high-fidelity, high-throughput, and safety-constrained low-level rearrangement benchmark for domestic settings, substantially improving experimental reproducibility and accelerating algorithm development.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

High-quality benchmarks are the foundation for embodied AI research, enabling significant advancements in long-horizon navigation, manipulation and rearrangement tasks. However, as frontier tasks in robotics get more advanced, they require faster simulation speed, more intricate test environments, and larger demonstration datasets. To this end, we present MS-HAB, a holistic benchmark for low-level manipulation and in-home object rearrangement. First, we provide a GPU-accelerated implementation of the Home Assistant Benchmark (HAB). We support realistic low-level control and achieve over 3x the speed of prior magical grasp implementations at a fraction of the GPU memory usage. Second, we train extensive reinforcement learning (RL) and imitation learning (IL) baselines for future work to compare against. Finally, we develop a rule-based trajectory filtering system to sample specific demonstrations from our RL policies which match predefined criteria for robot behavior and safety. Combining demonstration filtering with our fast environments enables efficient, controlled data generation at scale.

Problem

• Develops a benchmark for low-level manipulation tasks in home environments.
• Enhances simulation speed and reduces GPU memory usage for robotics tasks.
• Creates a system for filtering robot demonstrations to ensure safety and behavior criteria.

Innovation

• GPU-accelerated Home Assistant Benchmark implementation
• Extensive RL and IL baselines for comparison
• Rule-based trajectory filtering for safe demonstrations