No Need to Look! Locating and Grasping Objects by a Robot Arm Covered with Sensitive Skin

2025-08-25

AI Summary

This work addresses the challenge of object localization and grasping by robots under vision-deprived conditions. We propose a whole-body tactile-driven, multi-stage search-and-manipulation framework. Methodologically, we pioneer the use of a full-arm electronic skin as a distributed tactile sensing interface, integrating whole-body tactile feedback, end-effector six-axis force/torque measurements, and a coarse-to-fine search strategy: coarse exploration via large-scale arm-body contact narrows down the target region, followed by high-resolution end-effector tactile sensing for precise localization and grasping. Evaluations in both simulation and real-world platforms demonstrate an 85.7% single-object grasping success rate, a sixfold improvement in search speed over end-effector-only tactile baselines, and scalability to multi-object scenarios. This work breaks the conventional reliance on end-effector tactile sensing alone, establishing a new paradigm for robust robotic manipulation in low- or no-visual environments.

Technology Category

• Intelligent Robots: Manipulation
• Search and Optimization: Learning to Search
• Computer Vision: Low Level & Physics-based Vision

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

Locating and grasping of objects by robots is typically performed using visual sensors. Haptic feedback from contacts with the environment is only secondary if present at all. In this work, we explored an extreme case of searching for and grasping objects in complete absence of visual input, relying on haptic feedback only. The main novelty lies in the use of contacts over the complete surface of a robot manipulator covered with sensitive skin. The search is divided into two phases: (1) coarse workspace exploration with the complete robot surface, followed by (2) precise localization using the end-effector equipped with a force/torque sensor. We systematically evaluated this method in simulation and on the real robot, demonstrating that diverse objects can be located, grasped, and put in a basket. The overall success rate on the real robot for one object was 85.7% with failures mainly while grasping specific objects. The method using whole-body contacts is six times faster compared to a baseline that uses haptic feedback only on the end-effector. We also show locating and grasping multiple objects on the table. This method is not restricted to our specific setup and can be deployed on any platform with the ability of sensing contacts over the entire body surface. This work holds promise for diverse applications in areas with challenging visual perception (due to lighting, dust, smoke, occlusion) such as in agriculture when fruits or vegetables need to be located inside foliage and picked.

Problem

• Locating and grasping objects without visual input
• Using whole-body haptic feedback for object search
• Improving robotic manipulation in visually challenging environments

Innovation

• Whole-body sensitive skin for haptic feedback
• Two-phase search: coarse exploration then precise localization
• Force/torque sensor on end-effector for grasping