Hierarchical and Modular Network on Non-prehensile Manipulation in General Environments

2025-02-28

AI Summary

Non-grasping object manipulation (e.g., pushing, rolling) in generic environments like homes suffers from poor policy generalization due to geometric diversity—such as cabinet obstacles, step heights, and surface orientations. Method: We propose a modular, reconfigurable network architecture that extends the Contact-based Object Representation Network (CORN) into a unified geometric representation jointly encoding object and environment geometry. A procedural environment generation algorithm enables zero-shot transfer to unseen real-world scenes. Our approach integrates deep reinforcement learning, contact-aware geometric modeling, and digital twin simulation, achieving zero-shot cross-geometry transfer under full simulation training. Contributions/Results: (1) The first generalizable environment modeling framework tailored for non-grasping manipulation; (2) An open-source simulation benchmark comprising digital twins of nine real-world scenes and 353 diverse objects; (3) Significant improvement in policy transfer performance to previously unseen geometric environments.

Technology Category

• Intelligent Robots: Manipulation
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

For robots to operate in general environments like households, they must be able to perform non-prehensile manipulation actions such as toppling and rolling to manipulate ungraspable objects. However, prior works on non-prehensile manipulation cannot yet generalize across environments with diverse geometries. The main challenge lies in adapting to varying environmental constraints: within a cabinet, the robot must avoid walls and ceilings; to lift objects to the top of a step, the robot must account for the step's pose and extent. While deep reinforcement learning (RL) has demonstrated impressive success in non-prehensile manipulation, accounting for such variability presents a challenge for the generalist policy, as it must learn diverse strategies for each new combination of constraints. To address this, we propose a modular and reconfigurable architecture that adaptively reconfigures network modules based on task requirements. To capture the geometric variability in environments, we extend the contact-based object representation (CORN) to environment geometries, and propose a procedural algorithm for generating diverse environments to train our agent. Taken together, the resulting policy can zero-shot transfer to novel real-world environments and objects despite training entirely within a simulator. We additionally release a simulation-based benchmark featuring nine digital twins of real-world scenes with 353 objects to facilitate non-prehensile manipulation research in realistic domains.

Problem

• Generalizing non-prehensile manipulation across diverse environments
• Adapting to varying environmental constraints and geometries
• Enabling zero-shot transfer to novel real-world scenarios

Innovation

• Modular architecture adapts to task requirements
• Extended CORN for environmental geometric variability
• Simulator-trained policy transfers to real-world