HandFlow: Fully Generative 4D Hand Recovery with Flow Matching

2026-07-13

AI Summary

This work addresses the challenges of 4D hand reconstruction from monocular videos, which is highly susceptible to occlusions and motion blur, and where existing methods often lack temporal consistency and robustness. The authors propose a fully generative flow-matching framework that jointly denoises MANO parameters across an entire temporal window via a single ODE integration, enabling efficient and temporally coherent estimation of 3D hand pose and shape. The approach innovatively employs a Flux-style dual-stream Transformer to model long-range dependencies and introduces a confidence-aware continuous masking mechanism to handle missing or noisy observations, thereby circumventing autoregressive decoding. Evaluated on DexYCB and HOT3D, the method achieves state-of-the-art performance, reducing world-coordinate pose error by over 30%, attaining the lowest acceleration error, and reconstructing 150-frame sequences at 47 fps—approximately 12× faster than the fastest baseline.

Abstract

Accurate monocular 4D hand reconstruction remains challenging. Per-frame discriminative regressors lack temporal context and often produce jittery predictions. Temporal models improve consistency by aggregating information across frames, but they are typically deterministic regressors, making them vulnerable to ambiguous observations caused by occlusion and motion blur. Generative modeling offers a natural alternative by learning a prior over plausible hand motion sequences, enabling coherent hand-state recovery when visual evidence is incomplete or unreliable. Motivated by this observation, we present HandFlow, a fully generative flow-matching framework for temporally coherent 3D hand pose and shape estimation from monocular video. Given visual and skeletal observations, HandFlow denoises an entire temporal window of MANO parameters through a single ODE integration. To support this, we use a Flux-style dual-stream transformer that attends across the full sequence to capture long-range dependencies without autoregressive decoding, and a confidence-aware continuous masking mechanism that blends observed features with learnable mask tokens to handle noisy or missing observations. Experiments on DexYCB and HOT3D show that HandFlow achieves state-of-the-art performance, with particularly large gains in world-space accuracy and temporal smoothness. It reduces world-space pose error by over 30% compared with the strongest baseline and achieves the lowest acceleration error among all evaluated methods, while remaining competitive in per-frame pose accuracy. Moreover, on a single GPU HandFlow reconstructs a 150-frame sequence at 47 fps, about 12x faster than the fastest prior video-based method, with reconstruction itself accounting for only a small fraction of the end-to-end latency.

Problem

• 4D hand reconstruction
• temporal coherence
• occlusion
• motion blur
• monocular video

Innovation

• flow matching
• generative modeling
• temporal coherence
• dual-stream transformer
• confidence-aware masking