AirGlove: Exploring Egocentric 3D Hand Tracking and Appearance Generalization for Sensing Gloves

2026-02-05

AI Summary

Existing vision-based hand tracking models suffer significant performance degradation when users wear sensing gloves due to substantial appearance discrepancies. This work presents the first systematic evaluation of mainstream models’ zero-shot and fine-tuned performance across multiple sensing glove types. To address this challenge, we propose AirGlove, a novel approach that leverages cross-glove appearance representation learning combined with a data-efficient transfer strategy to enhance generalization to unseen gloves. Experimental results demonstrate that AirGlove substantially outperforms current methods across diverse sensing glove configurations, achieving marked improvements in both accuracy and robustness of 3D hand pose estimation.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Brain-Sensing and Analysis

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Sensing gloves have become important tools for teleoperation and robotic policy learning as they are able to provide rich signals like speed, acceleration and tactile feedback. A common approach to track gloved hands is to directly use the sensor signals (e.g., angular velocity, gravity orientation) to estimate 3D hand poses. However, sensor-based tracking can be restrictive in practice as the accuracy is often impacted by sensor signal and calibration quality. Recent advances in vision-based approaches have achieved strong performance on human hands via large-scale pre-training, but their performance on gloved hands with distinct visual appearances remains underexplored. In this work, we present the first systematic evaluation of vision-based hand tracking models on gloved hands under both zero-shot and fine-tuning setups. Our analysis shows that existing bare-hand models suffer from substantial performance degradation on sensing gloves due to large appearance gap between bare-hand and glove designs. We therefore propose AirGlove, which leverages existing gloves to generalize the learned glove representations towards new gloves with limited data. Experiments with multiple sensing gloves show that AirGlove effectively generalizes the hand pose models to new glove designs and achieves a significant performance boost over the compared schemes.

Problem

• hand tracking
• gloved hands
• appearance generalization
• vision-based sensing
• 3D hand pose

Innovation

• egocentric vision
• hand tracking
• appearance generalization
• sensing gloves
• zero-shot transfer