PhysV2A: Reachability-Gated and Semantic-Mask-Constrained Feasibility Completion for Video-to-Robot Manipulation

2026-07-10

AI Summary

This work addresses the challenge that 6D object motion priors inferred from videos are often agnostic to robot embodiment and thus difficult to execute directly. To bridge this gap, the authors propose a method that translates visual motion priors into robot-feasible manipulation trajectories by integrating trajectory-conditioned grasp feasibility assessment with full-motion planning. The approach leverages a vision-language model to generate semantic masks and incorporates hierarchical reachability gating, redundancy-aware priority optimization, and bounded Cartesian relaxation. This ensures both semantic consistency with the task intent and enhanced kinematic feasibility. Evaluated on four categories of tabletop manipulation tasks, the method significantly improves task success rates, substantially reduces infeasible trajectories, and produces manipulation paths with controllable semantic deviation and superior conditioning.

Abstract

Video-based manipulation provides object-centric motion priors from human demonstrations, generated videos, or RGB-D observations, but such priors are typically embodiment-agnostic and cannot be directly executed by a specific robot. This paper presents \textbf{PhysV2A}, a reachability-gated and semantic-mask-constrained feasibility-completion framework for converting video-derived 6D object motion into robot-executable manipulation trajectories. The key idea is to treat grasp feasibility as trajectory-conditioned rather than local: each RGB-D-generated 6-DoF grasp candidate is rigidly coupled with the recovered object motion to form a grasp-conditioned TCP trajectory hypothesis. PhysV2A then performs hierarchical reachability-gated selection, where infeasible grasp--trajectory pairs are rejected by robot-centric kinematic checks and surviving candidates are ranked by downstream execution suitability. For the selected reachable trajectory, a VLM-assisted and rule-validated S-Mask identifies task-critical and relaxable Cartesian components, enabling semantic-mask-constrained manipulability refinement through redundancy-first optimization and bounded Cartesian relaxation. Real-robot experiments on four tabletop manipulation tasks show that PhysV2A improves task success over representative video-prior and IK-only baselines, reduces kinematic-feasibility failures, and produces better-conditioned trajectories with bounded semantic deviations.

Problem

• video-to-robot manipulation
• grasp feasibility
• kinematic reachability
• trajectory completion
• embodiment gap

Innovation

• reachability-gated
• semantic-mask-constrained
• trajectory-conditioned grasp feasibility
• redundancy-first optimization
• video-to-robot manipulation