Unifying 3D Representation and Control of Diverse Robots with a Single Camera

2024-07-11

arXiv.org

AI Summary

Addressing the challenges of modeling and controlling soft, multi-material, uninstrumented heterogeneous robots, this paper introduces Neural Jacobian Fields—the first end-to-end vision-driven framework that requires no prior physical models, material assumptions, or manual annotations. Given only monocular video input, our method jointly learns 3D geometric representations and closed-loop motion policies via implicit neural fields coupled with differentiable Jacobian estimation, integrating self-supervised spatiotemporal consistency constraints and policy-gradient optimization. Evaluated across diverse robotic platforms—encompassing varied actuation mechanisms, material properties, and fabrication costs—the approach achieves millimeter-level pose accuracy, successfully disentangles and reconstructs the system’s dynamic topology, and enables zero-shot cross-platform policy transfer.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments

Abstract

Mirroring the complex structures and diverse functions of natural organisms is a long-standing challenge in robotics. Modern fabrication techniques have dramatically expanded feasible hardware, yet deploying these systems requires control software to translate desired motions into actuator commands. While conventional robots can easily be modeled as rigid links connected via joints, it remains an open challenge to model and control bio-inspired robots that are often multi-material or soft, lack sensing capabilities, and may change their material properties with use. Here, we introduce Neural Jacobian Fields, an architecture that autonomously learns to model and control robots from vision alone. Our approach makes no assumptions about the robot's materials, actuation, or sensing, requires only a single camera for control, and learns to control the robot without expert intervention by observing the execution of random commands. We demonstrate our method on a diverse set of robot manipulators, varying in actuation, materials, fabrication, and cost. Our approach achieves accurate closed-loop control and recovers the causal dynamic structure of each robot. By enabling robot control with a generic camera as the only sensor, we anticipate our work will dramatically broaden the design space of robotic systems and serve as a starting point for lowering the barrier to robotic automation.

Problem

• Modeling and controlling biologically inspired soft robots
• Inferring Jacobian fields from video for robot control
• Enabling robot control with a single camera sensor

Innovation

• Deep networks infer Jacobian fields from video
• Single-camera control without material assumptions
• Training via random commands without expert input