DexterCap: An Affordable and Automated System for Capturing Dexterous Hand-Object Manipulation

2026-01-09

AI Summary

Optical capture of fine-grained hand–object interactions remains highly challenging due to finger self-occlusion and subtle motion, with existing approaches either relying on expensive hardware or suffering from insufficient accuracy under occlusion. This work proposes DexterCap, a low-cost, fully automatic capture system that, for the first time, integrates densely patterned fiducial marker patches with an end-to-end automated reconstruction pipeline to achieve high accuracy and robustness in capturing hand–object interactions even under severe occlusion. Alongside the method, we introduce DexterHand, the first fine-grained dataset encompassing complex dexterous manipulations—such as Rubik’s Cube solving—featuring diverse objects and interaction behaviors. Both code and dataset are publicly released to advance research in this domain.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Humans and AI: Interaction Techniques and Devices

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance

Abstract

Capturing fine-grained hand-object interactions is challenging due to severe self-occlusion from closely spaced fingers and the subtlety of in-hand manipulation motions. Existing optical motion capture systems rely on expensive camera setups and extensive manual post-processing, while low-cost vision-based methods often suffer from reduced accuracy and reliability under occlusion. To address these challenges, we present DexterCap, a low-cost optical capture system for dexterous in-hand manipulation. DexterCap uses dense, character-coded marker patches to achieve robust tracking under severe self-occlusion, together with an automated reconstruction pipeline that requires minimal manual effort. With DexterCap, we introduce DexterHand, a dataset of fine-grained hand-object interactions covering diverse manipulation behaviors and objects, from simple primitives to complex articulated objects such as a Rubik's Cube. We release the dataset and code to support future research on dexterous hand-object interaction. Project website: https://pku-mocca.github.io/Dextercap-Page/

Problem

• hand-object interaction
• self-occlusion
• in-hand manipulation
• motion capture
• dexterous manipulation

Innovation

• character-coded markers
• self-occlusion robustness
• automated reconstruction
• dexterous manipulation
• hand-object interaction