Optimization of sim-to-real transfer in the humanoid robot NICO

2026-07-20

AI Summary

This work addresses the challenge of grasp failure in the humanoid robot NICO during sim-to-real transfer, primarily caused by localization errors. To overcome this without relying on RGB-D cameras or external tracking systems, the authors propose a lightweight grasping framework that integrates YOLO-based object and hand detection, fisheye stereo vision for 3D localization, a nonlinear tactile calibration model, and closed-loop visual feedback control into an end-to-end calibrated grasping pipeline. Experimental results demonstrate that the proposed nonlinear calibration model achieves the highest accuracy within the calibrated region, while the visual feedback mechanism substantially improves grasping success across the entire tabletop workspace, effectively bridging the sim-to-real gap.

Abstract

Robotic grasping requires accurate coordination between visual perception, object localization, inverse kinematics, and hand control. However, when movements planned in simulation are executed on a physical robot, the sim-to-real gap can cause small positioning errors that prevent successful grasping. In our previous work, we introduced a low-cost haptic calibration method that improved 2D reaching accuracy of the humanoid robot NICO. In this paper, we extend this approach from reaching to tabletop object grasping by adding YOLO-based object and hand detection, stereo vision-based localization using the robot's built-in low-resolution fisheye cameras, and task-specific corrections for grasp execution. Together, these components form a novel calibration-based grasping pipeline that does not require RGB-D cameras, motion capture, or external tracking systems. We also implemented a visual feedback model that aligns the robot hand with the detected object before grasping. Our results show that the fully nonlinear calibration model achieved the best performance inside the calibrated area, while the visual feedback model achieved the highest overall grasping success across the full tabletop workspace.

Problem

• sim-to-real transfer
• robotic grasping
• humanoid robot
• positioning error
• grasping success

Innovation

• sim-to-real transfer
• haptic calibration
• YOLO-based detection
• stereo vision localization
• visual feedback grasping