HOIDiNi: Human-Object Interaction through Diffusion Noise Optimization

2025-06-18

AI Summary

This work addresses three key challenges in text-driven 3D human–object interaction (HOI) motion generation: low contact accuracy, physically implausible motions, and insufficient motion diversity. We propose HOIDiNi—the first end-to-end framework generating realistic, physically plausible HOI motions directly in the noise space of a pre-trained diffusion model. Its core innovation is a two-stage noise-space optimization strategy: first, object-centric refinement to precisely localize hand–object contact points; second, human-centric optimization to ensure natural full-body kinematics. The framework integrates text-conditioned guidance, fine-tuning on the GRAB dataset, and explicit physical constraint modeling (e.g., collision avoidance and grasp stability). Quantitative and qualitative evaluations on GRAB demonstrate that HOIDiNi significantly outperforms prior methods in contact accuracy, physical plausibility, and visual quality. It robustly synthesizes complex interactions—including grasping, placing, and whole-body coordinated tasks—establishing new state-of-the-art performance.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Human-Robot Interaction
• Computer Vision: Diffusion Models for Vision

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We present HOIDiNi, a text-driven diffusion framework for synthesizing realistic and plausible human-object interaction (HOI). HOI generation is extremely challenging since it induces strict contact accuracies alongside a diverse motion manifold. While current literature trades off between realism and physical correctness, HOIDiNi optimizes directly in the noise space of a pretrained diffusion model using Diffusion Noise Optimization (DNO), achieving both. This is made feasible thanks to our observation that the problem can be separated into two phases: an object-centric phase, primarily making discrete choices of hand-object contact locations, and a human-centric phase that refines the full-body motion to realize this blueprint. This structured approach allows for precise hand-object contact without compromising motion naturalness. Quantitative, qualitative, and subjective evaluations on the GRAB dataset alone clearly indicate HOIDiNi outperforms prior works and baselines in contact accuracy, physical validity, and overall quality. Our results demonstrate the ability to generate complex, controllable interactions, including grasping, placing, and full-body coordination, driven solely by textual prompts. https://hoidini.github.io.

Problem

• Synthesizing realistic human-object interaction from text
• Balancing contact accuracy with natural motion diversity
• Optimizing diffusion noise for physical and visual plausibility

Innovation

• Optimizes diffusion noise for HOI synthesis
• Separates problem into object-centric and human-centric phases
• Generates interactions via textual prompts