MAD-HOI: Masked Autoregressive Diffusion for Generating Articulated Hand Object Interactions from Text

2026-08-10

AI Summary

This work addresses the limitations of existing text-driven hand–object interaction (HOI) generation methods, which struggle with variable-length composite action sequences, lack mechanisms for motion completion, in-filling, and reliable termination, and often compromise physical plausibility. To overcome these challenges, we propose MAD-HOI, a novel framework that uniquely integrates masked autoregressive modeling with flow matching generation. Our approach encodes hand and object motions in a disentangled continuous latent space and captures contextual dependencies via a masked autoregressive Transformer. This design enables unified support for generating both atomic and composite actions, performing conditional completion and in-filling, and predicting natural termination—all without requiring predefined sequence lengths and trained end-to-end with a single objective. Experiments on the ARCTIC and GRAB datasets demonstrate that our method outperforms existing open-source baselines in both motion diversity and physical realism.

Abstract

Methods for text-based generation of hand-object interaction (HOI) sequences primarily focus on producing smooth, physically plausible trajectories. A truly utilitarian method should additionally support variable-length generation, composite motion sequences, motion completion and infilling, and reliable termination without compromising physical plausibility. Standard diffusion models for HOI generation are typically trained only for text-to-motion generation on atomic motions and require the motion length to be specified a-priori. Autoregressive (AR) methods provide greater sequence-level flexibility, but commonly depend on discrete motion codes, which can lose contact-sensitive motion detail. To address these key limitations, we present a model performing Masked Autoregression with Diffusion for HOI generation (MAD-HOI). Our method starts by encoding hand and object motions in a continuous latent space while keeping them disentangled to maintain stream-wise control. This is followed by a masked autoregressive transformer to predict context features that condition a flow-matching head. MAD-HOI is capable of motion generation for atomic and composite articulated sequences, conditioned motion completion and infilling, as well as EOM (End of Motion) prediction from a single training objective. We provide comprehensive evaluations for these capabilities and benchmark our method on the ARCTIC and GRAB datasets. Our experiments demonstrate that our method generates more diverse and physically plausible interactions compared to other open-sourced baseline methods.

Problem

• hand-object interaction
• text-to-motion generation
• variable-length generation
• motion completion
• physical plausibility

Innovation

• Masked Autoregressive Diffusion
• Hand-Object Interaction
• Flow Matching
• Motion Infilling
• End-of-Motion Prediction