BimArt: A Unified Approach for the Synthesis of 3D Bimanual Interaction with Articulated Objects

2024-12-06

arXiv.org

AI Summary

This work addresses key limitations in generating 3D bimanual robotic manipulation animations of articulated objects—namely, reliance on reference grasp poses, rigid phase separation (grasping vs. manipulation), and coarse trajectory planning. We propose BimArt, an end-to-end framework that unifies bimanual motion and articulated object dynamics without requiring reference grasps or explicit phase segmentation. Our core contributions are: (1) a distance-field-based contact graph prior that explicitly encodes hand–object contact relationships; (2) a joint-aware feature representation jointly modeling object articulation structure and hand dynamics; and (3) a contact-graph-conditioned diffusion model for generating temporally coherent hand sequences. Quantitative and qualitative evaluations demonstrate that BimArt significantly outperforms state-of-the-art methods in motion plausibility, robustness to object geometry and kinematic constraints, and diversity of bimanual coordination patterns.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Computer Vision: Biometrics, Face, Gesture & Pose

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

We present BimArt, a novel generative approach for synthesizing 3D bimanual hand interactions with articulated objects. Unlike prior works, we do not rely on a reference grasp, a coarse hand trajectory, or separate modes for grasping and articulating. To achieve this, we first generate distance-based contact maps conditioned on the object trajectory with an articulation-aware feature representation, revealing rich bimanual patterns for manipulation. The learned contact prior is then used to guide our hand motion generator, producing diverse and realistic bimanual motions for object movement and articulation. Our work offers key insights into feature representation and contact prior for articulated objects, demonstrating their effectiveness in taming the complex, high-dimensional space of bimanual hand-object interactions. Through comprehensive quantitative experiments, we demonstrate a clear step towards simplified and high-quality hand-object animations that excel over the state-of-the-art in motion quality and diversity.

Problem

• Synthesizing 3D bimanual hand interactions with articulated objects
• Generating diverse and realistic bimanual motions for object manipulation
• Improving motion quality and diversity in hand-object animations

Innovation

• Generates contact maps for bimanual patterns
• Uses articulation-aware feature representation
• Guides motion with learned contact prior