Utilizing Inpainting for Keypoint Detection for Vision-Based Control of Robotic Manipulators

2026-04-14

AI Summary

This work proposes a model-free visual servoing framework based on image inpainting, enabling robust robot pose control using only natural features from the robot’s own body without reliance on external markers. The method leverages ArUco markers to automatically generate annotated training data that requires neither camera calibration nor an explicit robot model. During operation, image inpainting handles occlusions, while an unscented Kalman filter (UKF) is integrated to enhance temporal consistency of detected keypoints. The resulting system achieves stable and reliable vision-based control under both fully visible and partially occluded conditions, significantly improving the robustness of natural feature detection and tracking.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

In this paper we present a novel visual servoing framework to control a robotic manipulator in the configuration space by using purely natural visual features. Our goal is to develop methods that can robustly detect and track natural features or keypoints on robotic manipulators that would be used for vision-based control, especially for scenarios where placing external markers on the robot is not feasible or preferred at runtime. For the model training process of our data driven approach, we create a data collection pipeline where we attach ArUco markers along the robot's body, label their centers as keypoints, and then utilize an inpainting method to remove the markers and reconstruct the occluded regions. By doing so, we generate natural (markerless) robot images that are automatically labeled with the marker locations. These images are used to train a keypoint detection algorithm, which is used to control the robot configuration using natural features of the robot. Unlike the prior methods that rely on accurate camera calibration and robot models for labeling training images, our approach eliminates these dependencies through inpainting. To achieve robust keypoint detection even in the presence of occlusion, we introduce a second inpainting model, this time to utilize during runtime, that reconstructs occluded regions of the robot in real time, enabling continuous keypoint detection. To further enhance the consistency and robustness of keypoint predictions, we integrate an Unscented Kalman Filter (UKF) that refines the keypoint estimates over time, adding to stable and reliable control performance. We obtained successful control results with this model-free and purely vision-based control strategy, utilizing natural robot features in the runtime, both under full visibility and partial occlusion.

Problem

• keypoint detection
• vision-based control
• robotic manipulators
• occlusion
• markerless

Innovation

• inpainting
• keypoint detection
• visual servoing
• markerless tracking
• Unscented Kalman Filter