Latent Adaptive Planner for Dynamic Manipulation

2025-05-06

AI Summary

This work addresses the challenge of learning visuomotor policies from human video demonstrations for dynamic non-prehensile manipulation tasks. Methodologically, it formulates policy planning as a Bayesian inference process in a latent space, enabling temporally consistent online replanning via variational inference; introduces a model-driven scaling mapping mechanism to explicitly compensate for embodiment discrepancies between humans and robots; and establishes an end-to-end video-to-policy learning paradigm. Its core contribution is the first unified framework integrating latent-space Bayesian updating, adaptive replanning, and embodiment-aware mapping. Evaluated on multiple dynamic manipulation benchmarks, the approach achieves state-of-the-art performance: improving task success rate by 12.6%, trajectory smoothness by 23.4%, and energy efficiency by 18.9%, while supporting cross-robot deployment.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

This paper presents Latent Adaptive Planner (LAP), a novel approach for dynamic nonprehensile manipulation tasks that formulates planning as latent space inference, effectively learned from human demonstration videos. Our method addresses key challenges in visuomotor policy learning through a principled variational replanning framework that maintains temporal consistency while efficiently adapting to environmental changes. LAP employs Bayesian updating in latent space to incrementally refine plans as new observations become available, striking an optimal balance between computational efficiency and real-time adaptability. We bridge the embodiment gap between humans and robots through model-based proportional mapping that regenerates accurate kinematic-dynamic joint states and object positions from human demonstrations. Experimental evaluations across multiple complex manipulation benchmarks demonstrate that LAP achieves state-of-the-art performance, outperforming existing approaches in success rate, trajectory smoothness, and energy efficiency, particularly in dynamic adaptation scenarios. Our approach enables robots to perform complex interactions with human-like adaptability while providing an expandable framework applicable to diverse robotic platforms using the same human demonstration videos.

Problem

• Dynamic nonprehensile manipulation planning via latent space inference
• Visuomotor policy learning with variational replanning framework
• Bridging human-robot embodiment gap through kinematic-dynamic mapping

Innovation

• Latent space inference from human videos
• Variational replanning for temporal consistency
• Bayesian updating for real-time adaptability