Multimodal Priors-Augmented Text-Driven 3D Human-Object Interaction Generation

2026-02-11

AI Summary

This work addresses the challenges of unnatural human motion, distorted object dynamics, and weak interaction plausibility in text-driven 3D human-object interaction generation, which stem from the cross-modal gap between language and 3D motion. To overcome these limitations, we propose a generative framework that integrates priors from multimodal large language models. Our approach introduces a novel object representation combining geometric keypoints and contact-aware features, a modality-aware mixture-of-experts (MoE) fusion strategy, and a cascaded diffusion mechanism with explicit interaction supervision. Experimental results demonstrate that our method significantly outperforms existing approaches in generating high-fidelity, fine-grained 3D human-object interactions, effectively enhancing the naturalness of both human and object motion as well as the physical plausibility of their interactions.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

We address the challenging task of text-driven 3D human-object interaction (HOI) motion generation. Existing methods primarily rely on a direct text-to-HOI mapping, which suffers from three key limitations due to the significant cross-modality gap: (Q1) sub-optimal human motion, (Q2) unnatural object motion, and (Q3) weak interaction between humans and objects. To address these challenges, we propose MP-HOI, a novel framework grounded in four core insights: (1) Multimodal Data Priors: We leverage multimodal data (text, image, pose/object) from large multimodal models as priors to guide HOI generation, which tackles Q1 and Q2 in data modeling. (2) Enhanced Object Representation: We improve existing object representations by incorporating geometric keypoints, contact features, and dynamic properties, enabling expressive object representations, which tackles Q2 in data representation. (3) Multimodal-Aware Mixture-of-Experts (MoE) Model: We propose a modality-aware MoE model for effective multimodal feature fusion paradigm, which tackles Q1 and Q2 in feature fusion. (4) Cascaded Diffusion with Interaction Supervision: We design a cascaded diffusion framework that progressively refines human-object interaction features under dedicated supervision, which tackles Q3 in interaction refinement. Comprehensive experiments demonstrate that MP-HOI outperforms existing approaches in generating high-fidelity and fine-grained HOI motions.

Problem

• text-driven 3D generation
• human-object interaction
• cross-modality gap
• motion generation
• multimodal priors

Innovation

• Multimodal Priors
• Human-Object Interaction
• Mixture-of-Experts
• Cascaded Diffusion
• 3D Motion Generation