Robust Visual Servoing under Human Supervision for Assembly Tasks

2025-04-16

AI Summary

Robust grasping and high-precision placement by mobile manipulators in human-supervised modular construction remain challenging due to visual occlusion, camera pose uncertainty, and stringent geometric tolerances. Method: This paper proposes a novel framework integrating hybrid eye-in-hand/eye-to-hand visual servoing with adaptive control barrier functions (CBFs), the first to incorporate adaptive CBFs into visual servoing for active compensation of ±5° camera pose errors—ensuring persistent marker visibility. It further combines human-in-the-loop closed-loop control with 6-DoF motion planning to guarantee structural stability during assembly. Contribution/Results: Experimental validation on a real-world mobile manipulation platform demonstrates successful multi-module assembly: marker visibility reaches 98.3%, end-effector placement accuracy is <1.2 mm, and system robustness against environmental uncertainties and operator variability is significantly enhanced—enabling reliable deployment in dynamic, unstructured construction environments.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

We propose a framework enabling mobile manipulators to reliably complete pick-and-place tasks for assembling structures from construction blocks. The picking uses an eye-in-hand visual servoing controller for object tracking with Control Barrier Functions (CBFs) to ensure fiducial markers in the blocks remain visible. An additional robot with an eye-to-hand setup ensures precise placement, critical for structural stability. We integrate human-in-the-loop capabilities for flexibility and fault correction and analyze robustness to camera pose errors, proposing adapted barrier functions to handle them. Lastly, experiments validate the framework on 6-DoF mobile arms.

Problem

• Enable mobile manipulators to reliably assemble structures
• Ensure visibility of fiducial markers during picking
• Integrate human supervision for flexibility and fault correction

Innovation

• Eye-in-hand visual servoing with CBFs
• Eye-to-hand setup for precise placement
• Human-in-the-loop for fault correction