Manipulating Elasto-Plastic Objects With 3D Occupancy and Learning-Based Predictive Control

2025-05-22

AI Summary

Addressing challenges in elastoplastic object manipulation—including severe self-occlusion, difficulty in deformation representation, and complex dynamics—this work proposes the first end-to-end framework. Our method models continuous object deformation via 3D occupancy representations, introduces a shape-driven action initialization module to accelerate motion planning, and establishes the first RGB-to-occupancy supervised dataset and acquisition platform specifically for elastoplastic manipulation. The framework integrates a multi-view RGB-based occupancy prediction network (combining 3D CNNs and GNNs), a learned dynamics model, and model predictive control (MPC). Evaluated in simulation and on real robotic platforms, it enables precise shaping of materials such as clay toward arbitrary target geometries. Results show a 42% reduction in average shape error and a 3.1× speedup in planning time, significantly advancing the practicality and generalizability of autonomous elastoplastic object manipulation.

Technology Category

• Intelligent Robots: Manipulation
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Manipulating elasto-plastic objects remains a significant challenge due to severe self-occlusion, difficulties of representation, and complicated dynamics. This work proposes a novel framework for elasto-plastic object manipulation with a quasi-static assumption for motions, leveraging 3D occupancy to represent such objects, a learned dynamics model trained with 3D occupancy, and a learning-based predictive control algorithm to address these challenges effectively. We build a novel data collection platform to collect full spatial information and propose a pipeline for generating a 3D occupancy dataset. To infer the 3D occupancy during manipulation, an occupancy prediction network is trained with multiple RGB images supervised by the generated dataset. We design a deep neural network empowered by a 3D convolution neural network (CNN) and a graph neural network (GNN) to predict the complex deformation with the inferred 3D occupancy results. A learning-based predictive control algorithm is introduced to plan the robot actions, incorporating a novel shape-based action initialization module specifically designed to improve the planner efficiency. The proposed framework in this paper can successfully shape the elasto-plastic objects into a given goal shape and has been verified in various experiments both in simulation and the real world.

Problem

• Addressing self-occlusion and representation challenges in elasto-plastic object manipulation
• Developing a learning-based predictive control for complex deformation dynamics
• Creating a 3D occupancy framework for accurate object shaping and control

Innovation

• 3D occupancy representation for object modeling
• Deep neural network for deformation prediction
• Learning-based predictive control for action planning