Monocular Vision Based Control Framework for Grasping

2026-07-08

AI Summary

This work addresses the challenge of grasping diverse rigid and deformable objects in unstructured environments, where existing methods often rely on tactile sensing or specialized hardware, lacking a unified and sensor-efficient solution. The authors propose a unified grasping framework that operates solely with monocular RGB vision and a position-controlled gripper. By integrating open-vocabulary detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation, the system recovers object geometry and motion cues. Notably, it introduces a language-guided stiffness inference mechanism that leverages visual deformation proxies to adaptively select appropriate grasping strategies. Without requiring tactile feedback or object-specific models, the approach demonstrates robust performance on a Franka Emika robotic arm, successfully handling a variety of objects—including lettuce, cheese, and croissants—thereby validating its generality and effectiveness.

Abstract

Grasping in unstructured environments requires handling objects with widely different mechanical properties, from soft and deformable items to rigid everyday objects. Most existing approaches address these categories separately and often rely on tactile sensing, object-specific models, or specialized grippers. In this paper, we present a unified monocular vision-based grasping framework that targets both soft and rigid objects within a single control pipeline, using only RGB input and a position-controlled gripper. The proposed system combines open-vocabulary object detection, image segmentation, boundary-aware point assignment, real-time point tracking, and monocular depth estimation to recover object motion and geometry from visual observations. A key component of the framework is a language-based stiffness estimation model that infers an object's expected compliance from its semantic description and provides an object-level prior for selecting the grasping strategy before contact. For deformable objects, grasp adaptation is governed by a Procrustes-based dissimilarity measure computed from tracked keypoints, which acts as a visual proxy for deformation. For rigid objects, the gripper width is regulated through the scaling of tracked point distances. We validate the proposed method in real-world pick-and-place experiments on a Franka Emika Research 3 arm using objects with substantially different mechanical properties, including lettuce, fresh mozzarella cheese, croissants, paper towels, and hard plastic bottles. Results demonstrate that the framework achieves stable grasping across both soft and rigid objects using visual feedback alone, highlighting a practical, sensor-efficient, and generalizable approach for food handling and household manipulation.

Problem

• grasping
• unstructured environments
• soft objects
• rigid objects
• monocular vision

Innovation

• monocular vision
• unified grasping framework
• language-based stiffness estimation
• deformable object manipulation
• visual feedback control