IMAGIN-4D: Image-Guided Controllable Interaction Generation

2026-06-22

AI Summary

Existing methods struggle to precisely control fine-grained details such as grasping style, body pose, object pose, and contact relationships when generating human-object interactions. This work proposes a diffusion-based 4D human-object interaction generation framework that innovatively disentangles a single reference image into spatially and temporally separated conditioning signals: spatially extracting interaction state tokens and temporally performing dynamic region queries to enable frame-wise differentiated guidance. A character-aware multi-stream conditioning mechanism synergistically fuses textual, waypoint, and visual cues, enhanced by AdaLN modulation, cross-attention, and a motion-to-image rendering pipeline, significantly improving interaction control fidelity. Experiments demonstrate that the proposed method outperforms current baselines on the FBM and BEHAVE datasets, achieving more accurate fine-grained interaction synthesis while maintaining high-quality motion and precise path following.

Abstract

Generating human-object interactions (HOI) is central to character animation, robotics, AR/VR, and embodied AI. Recent HOI generation methods synthesize motion from text, object geometry, and sparse waypoints, controlling action semantics and object trajectories. However, these signals underspecify interaction: the same prompt and trajectory can produce different grasps, approach directions, body poses, object poses, contacts, and body-object layouts. We address this ambiguity with a reference image as a visual specification of the desired interaction snapshot. However, a single global image representation conflates distinct cues and conditions all frames on identical visual evidence. We therefore introduce IMAGIN-4D, a diffusion-based HOI generator that decomposes image conditioning spatio-temporally. For spatial conditioning, IMAGIN-4D extracts supervised interaction-state tokens for body pose, object pose, body-object contact, and spatial relationships at the depicted frame. For temporal conditioning, it computes frame-aware tokens by querying image patches per generated frame, allowing sequence segments to attend to different visual cues from the same image. To balance image, text, and waypoint cues, IMAGIN-4D uses role-aware conditioning: text, waypoints, and interaction-state tokens use separate AdaLN streams, while frame-aware visual tokens cross-attend with motion tokens. Since HOI motion datasets lack paired images, we build a synthetic motion-to-image rendering pipeline from FullBodyManipulation (FBM) and introduce an image-adherence metric to evaluate whether generated motions match the reference snapshot. Experiments on FBM and BEHAVE show that IMAGIN-4D improves fine-grained interaction control over single-token and uniformly image-conditioned baselines while preserving waypoint-following and motion quality. Code and models will be released at https://imagin4d.github.io.

Problem

• human-object interaction
• interaction ambiguity
• visual specification
• fine-grained control
• motion generation

Innovation

• image-conditioned generation
• spatio-temporal decomposition
• role-aware conditioning
• human-object interaction
• diffusion-based motion synthesis