Task-Aware Bimanual Affordance Prediction via VLM-Guided Semantic-Geometric Reasoning

2026-04-09

AI Summary

This work addresses the challenge of dual-arm manipulation, which requires simultaneous identification of task-relevant object affordances and assignment of left/right arm roles—a task at which purely geometric approaches often fail due to their lack of semantic understanding. The authors propose a hierarchical framework that fuses multi-view RGB-D data into a consistent 3D scene representation, generates 6-DoF grasp candidates, and—novelty introduced here—integrates a vision-language model (VLM) to perform task-level semantic reasoning. This enables joint optimization of semantic relevance and geometric feasibility without requiring category-specific training, allowing generalization across object categories and task descriptions. Evaluated on nine real-world dual-arm tasks spanning four categories, the method significantly outperforms purely geometric or semantic baselines, demonstrating the efficacy of semantic guidance for reliable dual-arm manipulation in unstructured environments.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Language and Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Bimanual manipulation requires reasoning about where to interact with an object and which arm should perform each action, a joint affordance localization and arm allocation problem that geometry-only planners cannot resolve without semantic understanding of task intent. Existing approaches either treat affordance prediction as coarse part segmentation or rely on geometric heuristics for arm assignment, failing to jointly reason about task-relevant contact regions and arm allocation. We reframe bimanual manipulation as a joint affordance localization and arm allocation problem and propose a hierarchical framework for task-aware bimanual affordance prediction that leverages a Vision-Language Model (VLM) to generalize across object categories and task descriptions without requiring category-specific training. Our approach fuses multi-view RGB-D observations into a consistent 3D scene representation and generates global 6-DoF grasp candidates, which are then spatially and semantically filtered by querying the VLM for task-relevant affordance regions on each object, as well as for arm allocation to the individual objects, thereby ensuring geometric validity while respecting task semantics. We evaluate our method on a dual-arm platform across nine real-world manipulation tasks spanning four categories: parallel manipulation, coordinated stabilization, tool use, and human handover. Our approach achieves consistently higher task success rates than geometric and semantic baselines for task-oriented grasping, demonstrating that explicit semantic reasoning over affordances and arm allocation helps enable reliable bimanual manipulation in unstructured environments.

Problem

• bimanual manipulation
• affordance prediction
• arm allocation
• task-aware reasoning
• semantic-geometric reasoning

Innovation

• bimanual manipulation
• affordance prediction
• Vision-Language Model (VLM)
• semantic-geometric reasoning
• task-aware grasping