Rapid Adaptation of Particle Dynamics for Generalized Deformable Object Mobile Manipulation

2026-03-18

AI Summary

This work addresses the significant impact of deformations—such as stretching and bending—on manipulation policies when operating on deformable objects with unknown dynamic parameters. The authors propose RAPiD, a novel approach that extends the Rapid Motor Adaptation (RMA) framework to deformable object manipulation for the first time. In simulation, a visuomotor policy conditioned on a dynamics embedding is learned using privileged information such as mass and particle positions. During real-world execution, the same embedding is inferred from non-privileged visual observations and action sequences, enabling effective policy transfer. By integrating particle-based object modeling, embedding encoding, and cross-domain adaptive inference, the system achieves end-to-end control on a 22-degree-of-freedom mobile manipulator, attaining over 80% success rates across two real-world tasks involving diverse object categories, instances, and dynamic properties.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Learning with Manifolds
• Computer Vision: Motion & Tracking

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

We address the challenge of learning to manipulate deformable objects with unknown dynamics. In non-rigid objects, the dynamics parameters define how they react to interactions -- how they stretch, bend, compress, and move -- and they are critical to determining the optimal actions to perform a manipulation task successfully. In other robotic domains, such as legged locomotion and in-hand rigid object manipulation, state-of-the-art approaches can handle unknown dynamics using Rapid Motor Adaptation (RMA). Through a supervised procedure in simulation that encodes each rigid object's dynamics, such as mass and position, these approaches learn a policy that conditions actions on a vector of latent dynamic parameters inferred from sequences of state-actions. However, in deformable object manipulation, the object's dynamics not only includes its mass and position, but also how the shape of the object changes. Our key insight is that the recent ground-truth particle positions of a deformable object in simulation capture changes in the object's shape, making it possible to extend RMA to deformable object manipulation. This key insight allows us to develop RAPiD, a two-phase method that learns to perform real-robot deformable object mobile manipulation by: 1) learning a visuomotor policy conditioned on the object's dynamics embedding, which is encoded from the object's privileged information in simulation, such as its mass and ground-truth particle positions, and 2) learning to infer this embedding using non-privileged information instead, such as robot visual observations and actions, so that the learned policy can transfer to the real world. On a mobile manipulator with 22 degrees of freedom, RAPiD enables over 80%+ success rates across two vision-based deformable object mobile manipulation tasks in the real world, under various object dynamics, categories, and instances.

Problem

• deformable object manipulation
• unknown dynamics
• mobile manipulation
• particle dynamics
• robotic adaptation

Innovation

• Rapid Motor Adaptation
• deformable object manipulation
• particle dynamics
• sim-to-real transfer
• visuomotor policy