DyWA: Dynamics-adaptive World Action Model for Generalizable Non-prehensile Manipulation

2025-03-21

AI Summary

Existing non-grasping manipulation methods rely on multi-view vision and precise pose tracking, exhibiting weak physical generalization and difficulty adapting to dynamic variations in object mass, friction, and other physical properties. This paper introduces the first dynamics-adaptive world action model for contact-rich manipulation, operating solely on single-view point cloud sequences. Our approach jointly models geometry, state, physics, and action: a point-cloud-based state prediction network estimates contact evolution; an implicit dynamics parameter encoding module enables online identification of mass and friction coefficients; and an end-to-end differentiable, physics-guided action decoder generates robust pushing and sliding policies. The method achieves robust manipulation of thin, large, and highly slippery objects. In simulation, success rates improve by 31.5%; on real hardware, the average success rate reaches 68%. We demonstrate challenging tasks including water bottle tipping and sliding on low-friction surfaces.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Low Level & Physics-based Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Nonprehensile manipulation is crucial for handling objects that are too thin, large, or otherwise ungraspable in unstructured environments. While conventional planning-based approaches struggle with complex contact modeling, learning-based methods have recently emerged as a promising alternative. However, existing learning-based approaches face two major limitations: they heavily rely on multi-view cameras and precise pose tracking, and they fail to generalize across varying physical conditions, such as changes in object mass and table friction. To address these challenges, we propose the Dynamics-Adaptive World Action Model (DyWA), a novel framework that enhances action learning by jointly predicting future states while adapting to dynamics variations based on historical trajectories. By unifying the modeling of geometry, state, physics, and robot actions, DyWA enables more robust policy learning under partial observability. Compared to baselines, our method improves the success rate by 31.5% using only single-view point cloud observations in the simulation. Furthermore, DyWA achieves an average success rate of 68% in real-world experiments, demonstrating its ability to generalize across diverse object geometries, adapt to varying table friction, and robustness in challenging scenarios such as half-filled water bottles and slippery surfaces.

Problem

• Overcoming reliance on multi-view cameras and precise pose tracking
• Generalizing across varying physical conditions like mass and friction
• Enhancing nonprehensile manipulation in unstructured environments

Innovation

• Dynamics-adaptive action model for manipulation
• Unifies geometry, state, physics, and actions
• Improves success rate with single-view observations