Observe Then Act: Asynchronous Active Vision-Action Model for Robotic Manipulation

2024-09-23

arXiv.org

AI Summary

To address robotic manipulation challenges under visual constraints—such as occlusions and narrow fields of view—this paper proposes an asynchronous active perception–action coordination framework that overcomes limitations of fixed camera viewpoints. The method jointly optimizes camera viewpoint selection and end-effector grasping pose via a novel task-driven, vision–action asynchronous serial strategy (Next-Best-View + Next-Best-Pose). It further enables cross-modal sensor–actuator coordination through few-shot reinforcement learning. Evaluated on eight visually constrained tasks from RLBench, the approach achieves significant improvements in task success rate and robustness compared to baseline methods. Experimental results demonstrate that active perception—specifically, dynamic viewpoint adaptation tightly coupled with action planning—provides critical performance gains for complex manipulation tasks under severe visual restrictions.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Vision for Robotics & Autonomous Driving
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

In real-world scenarios, many robotic manipulation tasks are hindered by occlusions and limited fields of view, posing significant challenges for passive observation-based models that rely on fixed or wrist-mounted cameras. In this paper, we investigate the problem of robotic manipulation under limited visual observation and propose a task-driven asynchronous active vision-action model.Our model serially connects a camera Next-Best-View (NBV) policy with a gripper Next-Best Pose (NBP) policy, and trains them in a sensor-motor coordination framework using few-shot reinforcement learning. This approach allows the agent to adjust a third-person camera to actively observe the environment based on the task goal, and subsequently infer the appropriate manipulation actions.We trained and evaluated our model on 8 viewpoint-constrained tasks in RLBench. The results demonstrate that our model consistently outperforms baseline algorithms, showcasing its effectiveness in handling visual constraints in manipulation tasks.

Problem

• Robotic manipulation with limited visual observation
• Handling occlusions and narrow fields of view
• Task-driven active vision-action model development

Innovation

• Active vision-action model
• Next-Best-View policy
• Few-shot reinforcement learning