HandOcc: NeRF-based Hand Rendering with Occupancy Networks

2025-05-04

AI Summary

Existing hand rendering methods rely on joint modeling of parametric meshes and Neural Radiance Fields (NeRF), suffering from limited mesh generalization, sensitivity to fitting errors, and resolution constraints. This paper proposes the first mesh-free neural hand rendering framework, which takes only a 3D skeletal pose as input and jointly models geometry and appearance via NeRF and an implicit occupancy field—enabling explicit decoupling of geometric structure and visual appearance. Key innovations include skeleton-conditioned rendering, a convolutional appearance encoder, and hand-hand interaction-aware occupancy modeling, eliminating dependence on predefined mesh templates. Evaluated on InterHand2.6M, our method achieves state-of-the-art performance: it significantly improves fine-grained hand detail fidelity, cross-pose appearance transfer quality, and physical plausibility of hand-hand interactions, while supporting real-time rendering and generalizable hand modeling across diverse poses and identities.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Manipulation

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We propose HandOcc, a novel framework for hand rendering based upon occupancy. Popular rendering methods such as NeRF are often combined with parametric meshes to provide deformable hand models. However, in doing so, such approaches present a trade-off between the fidelity of the mesh and the complexity and dimensionality of the parametric model. The simplicity of parametric mesh structures is appealing, but the underlying issue is that it binds methods to mesh initialization, making it unable to generalize to objects where a parametric model does not exist. It also means that estimation is tied to mesh resolution and the accuracy of mesh fitting. This paper presents a pipeline for meshless 3D rendering, which we apply to the hands. By providing only a 3D skeleton, the desired appearance is extracted via a convolutional model. We do this by exploiting a NeRF renderer conditioned upon an occupancy-based representation. The approach uses the hand occupancy to resolve hand-to-hand interactions further improving results, allowing fast rendering, and excellent hand appearance transfer. On the benchmark InterHand2.6M dataset, we achieved state-of-the-art results.

Problem

• Balancing fidelity and complexity in deformable hand models
• Overcoming limitations of mesh initialization and resolution
• Enabling meshless 3D hand rendering via occupancy networks

Innovation

• NeRF renderer with occupancy-based representation
• Meshless 3D rendering using 3D skeleton
• Hand occupancy resolves hand-to-hand interactions