Towards Exploratory and Focused Manipulation with Bimanual Active Perception: A New Problem, Benchmark and Strategy

2026-02-02

AI Summary

This work addresses the challenge of task-critical information loss caused by visual occlusion by proposing a novel paradigm of Exploratory and Focused Manipulation (EFM). We introduce active perception and dual-arm coordination into EFM for the first time, formally define the problem, and establish the EFM-10 benchmark comprising ten representative tasks along with the accompanying BAPData dataset. The proposed Bimanual Active Perception (BAP) strategy employs one arm for active visual exploration and the other for haptic-assisted manipulation, effectively integrating multimodal perception with coordinated control. Experimental results demonstrate that BAP significantly improves task success rates on the EFM-10 benchmark, providing the community with a reproducible foundation—including benchmarks, data, and methodological frameworks—for future research in this domain.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recently, active vision has reemerged as an important concept for manipulation, since visual occlusion occurs more frequently when main cameras are mounted on the robot heads. We reflect on the visual occlusion issue and identify its essence as the absence of information useful for task completion. Inspired by this, we come up with the more fundamental problem of Exploratory and Focused Manipulation (EFM). The proposed problem is about actively collecting information to complete challenging manipulation tasks that require exploration or focus. As an initial attempt to address this problem, we establish the EFM-10 benchmark that consists of 4 categories of tasks that align with our definition (10 tasks in total). We further come up with a Bimanual Active Perception (BAP) strategy, which leverages one arm to provide active vision and another arm to provide force sensing while manipulating. Based on this idea, we collect a dataset named BAPData for the tasks in EFM-10. With the dataset, we successfully verify the effectiveness of the BAP strategy in an imitation learning manner. We hope that the EFM-10 benchmark along with the BAP strategy can become a cornerstone that facilitates future research towards this direction. Project website: EFManipulation.github.io.

Problem

• Exploratory and Focused Manipulation
• active perception
• visual occlusion
• bimanual manipulation
• information acquisition

Innovation

• Exploratory and Focused Manipulation
• Bimanual Active Perception
• Active Vision
• Manipulation Benchmark
• Imitation Learning