Learning Dense Hand Contact Estimation from Imbalanced Data

2025-05-16

AI Summary

In hand contact estimation, two pervasive imbalances—class imbalance (contact vs. non-contact) and spatial imbalance (e.g., oversampling of fingertips and undersampling of palm/wrist regions)—severely degrade prediction accuracy. To jointly address both, this work proposes a novel framework: (1) a balanced contact sampling strategy to mitigate data distribution bias, and (2) a vertex-level class balancing (VCB) loss, the first to simultaneously optimize per-vertex class weights and spatial-frequency-adaptive weighting. Built upon a graph convolutional network, our method learns dense hand mesh representations and incorporates grouped resampling to enhance learning from low-frequency spatial regions. Evaluated on multiple large-scale hand interaction datasets, it achieves significant improvements in overall contact prediction accuracy, particularly for challenging areas such as the palm and wrist. The source code will be made publicly available.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Learning with Manifolds
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Hands are essential to human interaction, and understanding contact between hands and the world can promote comprehensive understanding of their function. Recently, there have been growing number of hand interaction datasets that cover interaction with object, other hand, scene, and body. Despite the significance of the task and increasing high-quality data, how to effectively learn dense hand contact estimation remains largely underexplored. There are two major challenges for learning dense hand contact estimation. First, there exists class imbalance issue from hand contact datasets where majority of samples are not in contact. Second, hand contact datasets contain spatial imbalance issue with most of hand contact exhibited in finger tips, resulting in challenges for generalization towards contacts in other hand regions. To tackle these issues, we present a framework that learns dense HAnd COntact estimation (HACO) from imbalanced data. To resolve the class imbalance issue, we introduce balanced contact sampling, which builds and samples from multiple sampling groups that fairly represent diverse contact statistics for both contact and non-contact samples. Moreover, to address the spatial imbalance issue, we propose vertex-level class-balanced (VCB) loss, which incorporates spatially varying contact distribution by separately reweighting loss contribution of each vertex based on its contact frequency across dataset. As a result, we effectively learn to predict dense hand contact estimation with large-scale hand contact data without suffering from class and spatial imbalance issue. The codes will be released.

Problem

• Learning dense hand contact estimation from imbalanced datasets
• Addressing class imbalance in hand contact data samples
• Resolving spatial imbalance in hand contact distribution

Innovation

• Balanced contact sampling for class imbalance
• Vertex-level class-balanced loss for spatial imbalance
• Framework for dense hand contact estimation